From 27b7b453a9871bb3f1432cff0db365accafcbfbb Mon Sep 17 00:00:00 2001 From: Jim Angel Date: Mon, 3 Dec 2018 19:21:11 -0600 Subject: [PATCH] Official 1.13 Release Docs (#11401) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Update metadata.generation behaviour for custom resources (#10705) * update docs promoting plugins to beta (#10796) * docs update to promote TaintBasedEvictions to beta (#10765) * First Korean l10n work for dev-1.13 (#10719) * Update outdated l10n(ko) contents (#10689) fixes #10686 * Translate concepts/overview/what-is-kubernetes in Korean (#10690) * Translate concepts/overview/what-is-kubernetes in Korean * Feedback from ClaudiaJKang * Translate concepts/overview/components in Korean (#10882) * Translate concepts/overview/components in Korean #10717 * Translate concepts/overview/components in Korean * Translate concepts/overview/components in Korean * Apply Korean glossary: 서비스 어카운트 * Translate concepts/overview/kubernetes-api in Korean (#10773) * Translate concepts/overview/kubernetes-api in Korean * Applied feedback from ianychoi * kubeadm: update the configuration docs to v1beta1 (#10959) * kubeadm: add small v1beta1 related updates (#10988) * ADD content/zh/docs/reference/setup-tools/kubeadm/kubeadm.md (#11031) * ADD content/zh/docs/reference/setup-tools/kubeadm/kubeadm.md * ADD content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md * Update content/zh/docs/reference/setup-tools/kubeadm/kubeadm.md Accepted Co-Authored-By: YouthLab * do not change 'master' or 'worker' nodes to '主从' * Doc updates for volume scheduling GA (#10743) * Doc updates for volume scheduling GA * Make trivial change to kick build * Document nodelease feature (#10699) * advanced audit doc for ModeBlockingStrict (#10203) * Rename EncryptionConfig to EncryptionConfiguration (#11080) EncryptionConfig was renamed to EncryptedConfiguration and added to the `apiserver.config.k8s.io` API group in Kubernetes 1.13. The feature was previously in alpha and was not handling versions properly, which lead to an originally unnoticed `v1` in the docs. * content/zh/docs/reference/setup-tools/kubeadm/kubeadm-init.md * trsanlate create-cluster-kubeadm.md to chinese (#11041) * trsanlate create-cluster-kubeadm.md to chinese * Update create-cluster-kubeadm.md * update the feature stage in v1.13 (#11307) * update new feature gates to document (#11295) * refresh controller role list on rbac description page (#11290) * node labeling restriction docs (#10944) * Update 1.13 docs for CSI GA (#10893) * dynamic audit documentation (#9947) * adds dynamic audit documentation * Copyedit for clarity See also inline question/s * Fix feature state shortcode * Update feature state * changes wording for dynamic audit flag behavior * Minor copyedit * fix dynamic audit yaml * adds api enablement command to dynamic audit docs * change ordering dynamic audit appears in * add references to dynamic audit in webhook backend * reword dynamic audit reference * updates stages field for audit sink object * changes audit sink api definition; rewords policy * kubeadm: remove kube-proxy workaround (#11162) * zh-trans content/en/docs/setup/independent/install-kubeadm.md (#11338) * zh-trans content/en/docs/setup/independent/install-kubeadm.md * Update install-kubeadm.md * Update dry run feature to beta (#11140) * vSphere volume raw block support doc update (#10932) * Add docs for Windows DNS configurations (#10036) * Update docs for fields allowed at root of CRD schema (#9973) * Add docs for Windows DNS configurations * add device monitoring documentation (#9945) * kubeadm: adds upgrade instructions for 1.13 (#11138) * kubeadm: adds upgrade instructions for 1.13 Signed-off-by: Chuck Ha * add minor copyedits Addressed a couple of copyedit comments a bit more cleanly. * kubeadm: add improvements to HA docs (#11094) * kubeadm: add information and diagrams for HA topologies * kubeadm: update HA doc with simplified steps * kubeadm: update HA doc with simplified steps * edit ha, add new topology topic, reorder by weight * troubleshoot markdown * fix more markdown, fix links * more markdown * more markdown * more markdown * changes after reviewer comments * add steps about Weave * update note about stacked topology * kubeadm external etcd HA upgrade 1.13 (#11364) * kubeadm external etcd HA upgrade 1.13 Signed-off-by: Ruben Orduz * Update stacked controlplane steps * kubeadm cert documentation (#11093) * kubeadm certificate API and CSR documentation * copyedits * fix typo * PR for diff docs (#10789) * Empty commit against dev-1.13 for diff documentation * Complete Declarative maangement with diff commands * Second Korean l10n work for dev-1.13. (#11030) * Update outdated l10n(ko) contents (#10915) * Translate main menu for l10n(ko) docs (#10916) * Translate tasks/run-application/horizontal-pod-autoscale-walkthrough (#10980) * Translate content/ko/docs/concepts/overview/working-with-objects/kubernetes-object in Korean #11104 (#11332) * Pick-right-solution page translates into Korean. (#11340) * ko-trans: add jd/..., sap/..., ebay/..., homeoffice/... (#11336) * Translate concept/workloads/pods/pod-overview.md (#11092) Co-authored-by: June Yi Co-authored-by: Jesang Myung Co-authored-by: zerobig <38598117+zer0big@users.noreply.github.com> Co-authored-by: Claudia J.Kang Co-authored-by: lIuDuI <1693291525@qq.com> Co-authored-by: Woojin Na(Eddie) * Rename encryption-at-rest related objects (#11059) EncryptionConfig was renamed to EncryptedConfiguration and added to the `apiserver.config.k8s.io` API group in Kubernetes 1.13. The feature was previously in alpha and was not handling versions properly, which lead to an originally unnoticed `v1` in the docs. Also, the `--experimental-encryption-provider-config` flag is now called just `--encryption-provider-config`. * Documenting FlexVolume Resize alpha feature. (#10097) * CR webhook conversion documentation (#10986) * CR Conversion * Addressing comments * Addressing more comments * Addressing even more comments * Addressing even^2 more comments * Remove references to etcd2 in v1.13 since support has been removed (#11414) * Remove etcd2 references as etcd2 is deprecated Link back to the v1.12 version of the etcd3 doc for the etcd2->etcd3 migration instructions. I updated the kube-apiserver reference manually, unsure if that is auto-generated somehow. The federation-apiserver can still potentially support etcd2 so I didn't touch that. * Remove outdated {master,node}.yaml files There are master/node yaml files that reference etcd2.service that are likely highly out of date. I couldn't find any docs that actually reference these templates so I removed them * Address review comments * Final Korean l10n work for dev-1.13 (#11440) * Update outdated l10n(ko) contents (#11425) fixes #11424 * Remove references to etcd2 in content/ko (#11416) * Resolve conflicts against master for /ko contents (#11438) * Fix unopened caution shortcode * kubeadm: update the reference docs for 1.13 (#10960) * docs update to promote TaintBasedEvictions to beta (#10765) * First Korean l10n work for dev-1.13 (#10719) * Update outdated l10n(ko) contents (#10689) fixes #10686 * Translate concepts/overview/what-is-kubernetes in Korean (#10690) * Translate concepts/overview/what-is-kubernetes in Korean * Feedback from ClaudiaJKang * Translate concepts/overview/components in Korean (#10882) * Translate concepts/overview/components in Korean #10717 * Translate concepts/overview/components in Korean * Translate concepts/overview/components in Korean * Apply Korean glossary: 서비스 어카운트 * Translate concepts/overview/kubernetes-api in Korean (#10773) * Translate concepts/overview/kubernetes-api in Korean * Applied feedback from ianychoi * kubeadm: update the configuration docs to v1beta1 (#10959) * kubeadm: add small v1beta1 related updates (#10988) * update new feature gates to document (#11295) * Update dry run feature to beta (#11140) * kubeadm: add improvements to HA docs (#11094) * kubeadm: add information and diagrams for HA topologies * kubeadm: update HA doc with simplified steps * kubeadm: update HA doc with simplified steps * edit ha, add new topology topic, reorder by weight * troubleshoot markdown * fix more markdown, fix links * more markdown * more markdown * more markdown * changes after reviewer comments * add steps about Weave * update note about stacked topology * kubeadm: update reference docs - add section about working with phases under kubeadm-init.md - update GA / beta status of features - kubeadm alpha phase was moved to kubeadm init phase - new commands were added under kubeadm alpha - included new CoreDNS usage examples * Generate components and tools reference * Add generated federation API Reference (#11491) * Add generated federation API Reference * Add front matter to federation reference * Remove whitespace from federation front matter * Remove more whitespace from federation front matter * Remove superfluous kubefed reference * Add frontmatter to generated kubefed reference * Fix kubefed reference page frontmatter * Generate kubectl reference docs 1.13 (#11487) * Generate kubectl reference docs 1.13 * Fix links in kubectl reference * Add 1.13 API reference (#11489) * Update config.toml (#11486) * Update config.toml Preparing for 1.13 release, updating the config.toml and dropping the 1.8 docs reference. * update dot releases and docsbranch typo * adding .Site. to Params.currentUrl (#11503) see https://github.com/kubernetes/website/pull/11502 for context * Add 1.13 Release notes (#11499) --- config.toml | 41 +- .../en/docs/concepts/architecture/nodes.md | 21 +- .../cluster-administration/cloud-providers.md | 36 +- .../concepts/configuration/assign-pod-node.md | 15 + .../configuration/taint-and-toleration.md | 14 +- .../compute-storage-net/device-plugins.md | 30 + .../declarative-config.md | 15 +- .../object-management-kubectl/overview.md | 9 +- .../concepts/storage/persistent-volumes.md | 13 +- .../docs/concepts/storage/storage-classes.md | 13 - content/en/docs/concepts/storage/volumes.md | 21 +- .../workloads/controllers/daemonset.md | 16 +- .../getting-started-guides/windows/_index.md | 30 +- content/en/docs/reference/_index.md | 1 + .../admission-controllers.md | 19 + .../docs/reference/access-authn-authz/rbac.md | 3 + .../cloud-controller-manager.md | 74 +- .../feature-gates.md | 35 +- .../federation-apiserver.md | 4 +- .../federation-controller-manager.md | 4 +- .../kube-apiserver.md | 140 +- .../kube-controller-manager.md | 83 +- .../kube-proxy.md | 2 +- .../kube-scheduler.md | 219 +- .../command-line-tools-reference/kubelet.md | 2 +- .../extensions/v1beta1/definitions.html | 4 +- .../extensions/v1beta1/operations.html | 2 +- .../federation/v1beta1/definitions.html | 1606 + .../federation/v1beta1/operations.html | 1921 + .../reference/federation/v1/definitions.html | 6 +- .../reference/federation/v1/operations.html | 2 +- content/en/docs/reference/kubectl/kubectl.md | 41 +- content/en/docs/reference/kubectl/overview.md | 11 +- .../en/docs/reference/kubernetes-api/index.md | 4 +- .../setup-tools/kubeadm/generated/kubeadm.md | 17 +- .../kubeadm/generated/kubeadm_alpha.md | 4 +- .../kubeadm/generated/kubeadm_alpha_certs.md | 50 + ..._renew.md => kubeadm_alpha_certs_renew.md} | 2 +- ...ll.md => kubeadm_alpha_certs_renew_all.md} | 20 +- ...lpha_certs_renew_apiserver-etcd-client.md} | 20 +- ...a_certs_renew_apiserver-kubelet-client.md} | 20 +- ...=> kubeadm_alpha_certs_renew_apiserver.md} | 24 +- ...ha_certs_renew_etcd-healthcheck-client.md} | 20 +- ...=> kubeadm_alpha_certs_renew_etcd-peer.md} | 20 +- ... kubeadm_alpha_certs_renew_etcd-server.md} | 20 +- ...m_alpha_certs_renew_front-proxy-client.md} | 20 +- .../generated/kubeadm_alpha_kubeconfig.md | 52 + ...er.md => kubeadm_alpha_kubeconfig_user.md} | 13 +- ...se_kubelet.md => kubeadm_alpha_kubelet.md} | 2 +- ...fig.md => kubeadm_alpha_kubelet_config.md} | 2 +- ... kubeadm_alpha_kubelet_config_download.md} | 4 +- ...dm_alpha_kubelet_config_enable-dynamic.md} | 4 +- ...kubeadm_alpha_phase_bootstrap-token_all.md | 122 - ...lpha_phase_bootstrap-token_cluster-info.md | 65 - ...eadm_alpha_phase_bootstrap-token_create.md | 114 - ...bootstrap-token_node_allow-auto-approve.md | 65 - ...se_bootstrap-token_node_allow-post-csrs.md | 65 - .../kubeadm_alpha_phase_certs_all.md | 111 - .../kubeadm_alpha_phase_kubeconfig_all.md | 109 - ...lpha_phase_kubelet_config_write-to-disk.md | 70 - ...eadm_alpha_phase_kubelet_write-env-file.md | 73 - .../kubeadm_alpha_phase_preflight.md | 64 - ...ase_etcd.md => kubeadm_alpha_preflight.md} | 4 +- ...ode.md => kubeadm_alpha_preflight_node.md} | 32 +- ...osting.md => kubeadm_alpha_selfhosting.md} | 0 ....md => kubeadm_alpha_selfhosting_pivot.md} | 23 +- .../kubeadm/generated/kubeadm_config.md | 2 +- .../generated/kubeadm_config_images.md | 2 +- .../generated/kubeadm_config_images_list.md | 4 +- .../generated/kubeadm_config_images_pull.md | 4 +- .../generated/kubeadm_config_migrate.md | 6 +- ...-token_node.md => kubeadm_config_print.md} | 12 +- ... => kubeadm_config_print_init-defaults.md} | 17 +- ... => kubeadm_config_print_join-defaults.md} | 38 +- .../generated/kubeadm_config_upload.md | 2 +- .../kubeadm_config_upload_from-file.md | 2 +- .../kubeadm_config_upload_from-flags.md | 4 +- .../kubeadm/generated/kubeadm_config_view.md | 2 +- .../kubeadm/generated/kubeadm_init.md | 54 +- .../kubeadm/generated/kubeadm_init_phase.md | 50 + ...e_addon.md => kubeadm_init_phase_addon.md} | 4 + ...all.md => kubeadm_init_phase_addon_all.md} | 35 +- ...md => kubeadm_init_phase_addon_coredns.md} | 18 +- ...=> kubeadm_init_phase_addon_kube-proxy.md} | 18 +- .../kubeadm_init_phase_bootstrap-token.md | 85 + ...e_certs.md => kubeadm_init_phase_certs.md} | 6 +- .../generated/kubeadm_init_phase_certs_all.md | 96 + ...init_phase_certs_apiserver-etcd-client.md} | 20 +- ...t_phase_certs_apiserver-kubelet-client.md} | 20 +- ... => kubeadm_init_phase_certs_apiserver.md} | 32 +- ...s_ca.md => kubeadm_init_phase_certs_ca.md} | 10 +- ...md => kubeadm_init_phase_certs_etcd-ca.md} | 6 +- ...it_phase_certs_etcd-healthcheck-client.md} | 20 +- ... => kubeadm_init_phase_certs_etcd-peer.md} | 20 +- ...> kubeadm_init_phase_certs_etcd-server.md} | 22 +- ...ubeadm_init_phase_certs_front-proxy-ca.md} | 6 +- ...dm_init_phase_certs_front-proxy-client.md} | 20 +- ...s_sa.md => kubeadm_init_phase_certs_sa.md} | 16 +- ...md => kubeadm_init_phase_control-plane.md} | 6 +- ...> kubeadm_init_phase_control-plane_all.md} | 42 +- ...adm_init_phase_control-plane_apiserver.md} | 29 +- ...phase_control-plane_controller-manager.md} | 23 +- ...adm_init_phase_control-plane_scheduler.md} | 21 +- .../generated/kubeadm_init_phase_etcd.md | 54 + ...al.md => kubeadm_init_phase_etcd_local.md} | 28 +- ...ig.md => kubeadm_init_phase_kubeconfig.md} | 4 + ...=> kubeadm_init_phase_kubeconfig_admin.md} | 16 +- .../kubeadm_init_phase_kubeconfig_all.md | 96 + ...it_phase_kubeconfig_controller-manager.md} | 16 +- ... kubeadm_init_phase_kubeconfig_kubelet.md} | 22 +- ...ubeadm_init_phase_kubeconfig_scheduler.md} | 16 +- ...md => kubeadm_init_phase_kubelet-start.md} | 35 +- .../kubeadm_init_phase_mark-control-plane.md | 78 + ...ter.md => kubeadm_init_phase_preflight.md} | 40 +- ...md => kubeadm_init_phase_upload-config.md} | 8 +- ...> kubeadm_init_phase_upload-config_all.md} | 19 +- ...beadm_init_phase_upload-config_kubeadm.md} | 17 +- ...beadm_init_phase_upload-config_kubelet.md} | 16 +- .../kubeadm/generated/kubeadm_join.md | 22 +- .../kubeadm/generated/kubeadm_reset.md | 7 + .../kubeadm/generated/kubeadm_token.md | 2 +- .../kubeadm/generated/kubeadm_token_create.md | 2 +- .../kubeadm/generated/kubeadm_token_delete.md | 2 +- .../generated/kubeadm_token_generate.md | 2 +- .../kubeadm/generated/kubeadm_token_list.md | 2 +- .../generated/kubeadm_upgrade_apply.md | 8 +- .../kubeadm/generated/kubeadm_upgrade_diff.md | 2 +- .../generated/kubeadm_upgrade_node_config.md | 6 +- ...upgrade_node_experimental-control-plane.md | 13 +- .../kubeadm/generated/kubeadm_upgrade_plan.md | 8 +- .../kubeadm/implementation-details.md | 56 +- .../setup-tools/kubeadm/kubeadm-alpha.md | 162 +- .../setup-tools/kubeadm/kubeadm-config.md | 11 +- .../setup-tools/kubeadm/kubeadm-init-phase.md | 155 + .../setup-tools/kubeadm/kubeadm-init.md | 96 +- .../setup-tools/kubeadm/kubeadm-join.md | 2 +- .../setup-tools/kubeadm/kubeadm-reset.md | 2 +- .../setup-tools/kubeadm/kubeadm-token.md | 2 +- .../setup-tools/kubeadm/kubeadm-upgrade.md | 3 + .../reference/setup-tools/kubeadm/kubeadm.md | 2 +- .../setup-tools/kubefed/kubefed-init.md | 105 - .../setup-tools/kubefed/kubefed-join.md | 99 - .../setup-tools/kubefed/kubefed-options.md | 77 - .../setup-tools/kubefed/kubefed-unjoin.md | 86 - .../setup-tools/kubefed/kubefed-version.md | 80 - .../reference/setup-tools/kubefed/kubefed.md | 10 +- .../setup-tools/kubefed/kubefed_init.md | 10 +- .../setup-tools/kubefed/kubefed_join.md | 10 +- .../setup-tools/kubefed/kubefed_options.md | 10 +- .../setup-tools/kubefed/kubefed_unjoin.md | 10 +- .../setup-tools/kubefed/kubefed_version.md | 10 +- .../docs/reference/using-api/api-concepts.md | 20 +- .../en/docs/setup/custom-cloud/master.yaml | 142 - content/en/docs/setup/custom-cloud/node.yaml | 92 - .../setup/independent/control-plane-flags.md | 60 +- .../independent/create-cluster-kubeadm.md | 28 +- .../en/docs/setup/independent/ha-topology.md | 71 + .../setup/independent/high-availability.md | 544 +- .../setup/independent/kubelet-integration.md | 2 +- .../independent/setup-ha-etcd-with-kubeadm.md | 40 +- .../independent/troubleshooting-kubeadm.md | 44 +- content/en/docs/setup/release/notes.md | 1890 +- .../custom-resource-definition-versioning.md | 153 +- .../custom-resource-definitions.md | 8 +- .../configure-upgrade-etcd.md | 234 +- .../docs/tasks/administer-cluster/coredns.md | 8 +- .../tasks/administer-cluster/encrypt-data.md | 27 +- .../tasks/administer-cluster/kms-provider.md | 18 +- .../kubeadm/kubeadm-certs.md | 124 + .../kubeadm/kubeadm-upgrade-1-13.md | 278 + .../kubeadm/kubeadm-upgrade-ha-1-12.md | 2 +- .../kubeadm/kubeadm-upgrade-ha-1-13.md | 168 + .../tasks/debug-application-cluster/audit.md | 54 + .../tasks/extend-kubectl/kubectl-plugins.md | 4 +- content/ko/case-studies/_index.html | 6 +- .../adform/adform_featured_logo.png | Bin 0 -> 7830 bytes content/ko/case-studies/adform/index.html | 118 + .../case-studies/amadeus/amadeus_featured.png | Bin 0 -> 6970 bytes .../ko/case-studies/amadeus/amadeus_logo.png | Bin 0 -> 19562 bytes content/ko/case-studies/amadeus/index.html | 105 + .../ancestry/ancestry_featured.png | Bin 0 -> 21016 bytes .../case-studies/ancestry/ancestry_logo.png | Bin 0 -> 21141 bytes content/ko/case-studies/ancestry/index.html | 117 + .../blablacar/blablacar_featured.png | Bin 0 -> 7867 bytes .../case-studies/blablacar/blablacar_logo.png | Bin 0 -> 7841 bytes content/ko/case-studies/blablacar/index.html | 98 + .../blackrock/blackrock_featured.png | Bin 0 -> 6391 bytes .../case-studies/blackrock/blackrock_logo.png | Bin 0 -> 6430 bytes content/ko/case-studies/blackrock/index.html | 112 + content/ko/case-studies/box/box_featured.png | Bin 0 -> 6348 bytes content/ko/case-studies/box/box_logo.png | Bin 0 -> 5988 bytes content/ko/case-studies/box/box_small.png | Bin 0 -> 8519 bytes content/ko/case-studies/box/index.html | 114 + content/ko/case-studies/box/video.png | Bin 0 -> 134595 bytes .../case-studies/buffer/buffer_featured.png | Bin 0 -> 5263 bytes .../ko/case-studies/buffer/buffer_logo.png | Bin 0 -> 4791 bytes content/ko/case-studies/buffer/index.html | 112 + .../capital-one/capitalone_featured_logo.png | Bin 0 -> 11306 bytes .../ko/case-studies/capital-one/index.html | 96 + .../ko/case-studies/ccp-games/ccp_logo.png | Bin 0 -> 20368 bytes content/ko/case-studies/ccp-games/index.html | 4 + .../ko/case-studies/comcast/comcast_logo.png | Bin 0 -> 6279 bytes content/ko/case-studies/comcast/index.html | 4 + .../concur/concur_featured_logo.png | Bin 0 -> 19719 bytes content/ko/case-studies/concur/index.html | 4 + .../crowdfire/crowdfire_featured_logo.png | Bin 0 -> 7827 bytes content/ko/case-studies/crowdfire/index.html | 101 + .../ko/case-studies/ebay/ebay_featured.png | Bin 0 -> 22471 bytes content/ko/case-studies/ebay/ebay_logo.png | Bin 0 -> 7316 bytes content/ko/case-studies/ebay/index.html | 4 + .../ko/case-studies/goldman-sachs/gs_logo.png | Bin 0 -> 23083 bytes .../ko/case-studies/goldman-sachs/index.html | 4 + .../case-studies/golfnow/golfnow_featured.png | Bin 0 -> 22915 bytes .../ko/case-studies/golfnow/golfnow_logo.png | Bin 0 -> 8858 bytes content/ko/case-studies/golfnow/index.html | 125 + .../haufegroup/haufegroup_featured.png | Bin 0 -> 5841 bytes .../haufegroup/haufegroup_logo.png | Bin 0 -> 5794 bytes content/ko/case-studies/haufegroup/index.html | 112 + .../homeoffice/homeoffice_logo.png | Bin 0 -> 27411 bytes content/ko/case-studies/homeoffice/index.html | 4 + .../case-studies/huawei/huawei_featured.png | Bin 0 -> 14310 bytes .../ko/case-studies/huawei/huawei_logo.png | Bin 0 -> 14274 bytes content/ko/case-studies/huawei/index.html | 101 + .../ko/case-studies/ibm/ibm_featured_logo.png | Bin 0 -> 8543 bytes .../ko/case-studies/ibm/ibm_featured_logo.svg | 1 + content/ko/case-studies/ibm/index.html | 111 + content/ko/case-studies/ing/index.html | 99 + .../ko/case-studies/ing/ing_featured_logo.png | Bin 0 -> 10305 bytes content/ko/case-studies/jd/index.html | 4 + content/ko/case-studies/jd/jd_logo.png | Bin 0 -> 10738 bytes content/ko/case-studies/liveperson/index.html | 4 + .../liveperson/liveperson_logo.png | Bin 0 -> 19458 bytes content/ko/case-studies/monzo/index.html | 4 + content/ko/case-studies/monzo/monzo_logo.png | Bin 0 -> 5515 bytes content/ko/case-studies/naic/index.html | 116 + .../case-studies/naic/naic_featured_logo.png | Bin 0 -> 9433 bytes .../ko/case-studies/newyorktimes/index.html | 108 + .../newyorktimes/newyorktimes_featured.png | Bin 0 -> 20189 bytes .../newyorktimes/newyorktimes_logo.png | Bin 0 -> 10251 bytes content/ko/case-studies/nordstrom/index.html | 110 + .../nordstrom/nordstrom_featured_logo.png | Bin 0 -> 7352 bytes .../northwestern-mutual/index.html | 99 + .../northwestern_featured_logo.png | Bin 0 -> 10083 bytes content/ko/case-studies/ocado/index.html | 99 + .../ocado/ocado_featured_logo.png | Bin 0 -> 8089 bytes content/ko/case-studies/openAI/index.html | 99 + .../case-studies/openAI/openai_featured.png | Bin 0 -> 12132 bytes .../ko/case-studies/openAI/openai_logo.png | Bin 0 -> 19818 bytes content/ko/case-studies/peardeck/index.html | 111 + .../peardeck/peardeck_featured.png | Bin 0 -> 11786 bytes .../case-studies/peardeck/peardeck_logo.png | Bin 0 -> 9260 bytes content/ko/case-studies/pearson/index.html | 87 + .../case-studies/pearson/pearson_featured.png | Bin 0 -> 7784 bytes .../ko/case-studies/pearson/pearson_logo.png | Bin 0 -> 5123 bytes content/ko/case-studies/philips/index.html | 4 + .../ko/case-studies/philips/philips_logo.png | Bin 0 -> 3877 bytes content/ko/case-studies/pinterest/index.html | 109 + .../pinterest/pinterest_feature.png | Bin 0 -> 9118 bytes .../case-studies/pinterest/pinterest_logo.png | Bin 0 -> 9003 bytes content/ko/case-studies/pokemon-go/index.html | 4 + .../pokemon-go/pokemon_go_logo.png | Bin 0 -> 19218 bytes .../ko/case-studies/samsung-sds/index.html | 4 + .../ko/case-studies/samsung-sds/sds_logo.png | Bin 0 -> 22930 bytes content/ko/case-studies/sap/index.html | 4 + content/ko/case-studies/sap/sap_logo.png | Bin 0 -> 20435 bytes content/ko/case-studies/sap/sap_small.png | Bin 0 -> 4879 bytes content/ko/case-studies/slingtv/index.html | 110 + .../slingtv/slingtv_featured_logo.png | Bin 0 -> 8513 bytes content/ko/case-studies/soundcloud/index.html | 4 + .../soundcloud/soundcloud_logo.png | Bin 0 -> 22800 bytes .../ko/case-studies/squarespace/index.html | 101 + .../squarespace/squarespace_featured_logo.png | Bin 0 -> 4539 bytes content/ko/case-studies/wepay/index.html | 4 + content/ko/case-studies/wepay/wepay_logo.png | Bin 0 -> 20277 bytes content/ko/case-studies/wikimedia/index.html | 96 + .../wikimedia/wikimedia_featured.png | Bin 0 -> 21322 bytes .../case-studies/wikimedia/wikimedia_logo.png | Bin 0 -> 7875 bytes content/ko/case-studies/wink/index.html | 109 + .../ko/case-studies/wink/wink_featured.png | Bin 0 -> 21292 bytes content/ko/case-studies/wink/wink_logo.png | Bin 0 -> 5623 bytes content/ko/case-studies/workiva/index.html | 113 + .../workiva/workiva_featured_logo.png | Bin 0 -> 5980 bytes .../ko/case-studies/yahoo-japan/index.html | 4 + .../yahoo-japan/yahooJapan_logo.png | Bin 0 -> 7266 bytes content/ko/case-studies/ygrene/index.html | 111 + .../ygrene/ygrene_featured_logo.png | Bin 0 -> 11569 bytes content/ko/case-studies/zalando/index.html | 101 + .../zalando/zalando_feature_logo.png | Bin 0 -> 7643 bytes content/ko/case-studies/zulily/index.html | 4 + .../case-studies/zulily/zulily_featured.png | Bin 0 -> 7953 bytes .../ko/case-studies/zulily/zulily_logo.png | Bin 0 -> 8815 bytes content/ko/docs/_index.md | 3 + .../ko/docs/concepts/overview/components.md | 105 + .../docs/concepts/overview/kubernetes-api.md | 128 + .../concepts/overview/what-is-kubernetes.md | 301 +- .../overview/working-with-objects/_index.md | 4 + .../kubernetes-objects.md | 69 + content/ko/docs/concepts/workloads/_index.md | 4 + .../ko/docs/concepts/workloads/pods/_index.md | 4 + .../concepts/workloads/pods/pod-overview.md | 107 + content/ko/docs/home/_index.md | 2 +- content/ko/docs/reference/_index.md | 58 + .../ko/docs/reference/glossary/controller.md | 19 + content/ko/docs/reference/glossary/etcd.md | 19 + .../docs/reference/glossary/kube-apiserver.md | 19 + .../glossary/kube-controller-manager.md | 19 + .../docs/reference/glossary/kube-scheduler.md | 18 + content/ko/docs/reference/glossary/kubelet.md | 19 + content/ko/docs/setup/minikube.md | 40 +- content/ko/docs/setup/multiple-zones.md | 30 +- content/ko/docs/setup/scratch.md | 2 +- content/ko/docs/tasks/_index.md | 87 + .../ko/docs/tasks/run-application/_index.md | 4 + .../horizontal-pod-autoscale-walkthrough.md | 375 + content/ko/docs/tutorials/hello-minikube.md | 35 +- .../deploy-app/deploy-intro.html | 11 +- .../update/update-intro.html | 1 + .../ko/examples/application/deployment.yaml | 19 + .../examples/application/hpa/php-apache.yaml | 13 + content/ko/examples/minikube/Dockerfile | 2 +- .../kubeadm/generated/kubeadm_init.md | 203 + .../setup-tools/kubeadm/kubeadm-init.md | 529 + .../reference/setup-tools/kubeadm/kubeadm.md | 33 + .../independent/create-cluster-kubeadm.md | 1115 + .../docs/setup/independent/install-kubeadm.md | 397 + .../tasks/administer-cluster/encrypt-data.md | 12 +- layouts/shortcodes/deprecationwarning.html | 2 +- .../generated/kubectl/kubectl-commands.html | 545 +- .../reference/generated/kubectl/navData.js | 2 +- .../node_modules/jquery/dist/jquery.min.js | 6 +- .../v1.13/css/bootstrap.min.css | 6 + .../v1.13/css/font-awesome.min.css | 4 + .../kubernetes-api/v1.13/css/stylesheet.css | 228 + .../v1.13/fonts/FontAwesome.otf | Bin 0 -> 134808 bytes .../v1.13/fonts/fontawesome-webfont.eot | Bin 0 -> 165742 bytes .../v1.13/fonts/fontawesome-webfont.svg | 2671 + .../v1.13/fonts/fontawesome-webfont.ttf | Bin 0 -> 165548 bytes .../v1.13/fonts/fontawesome-webfont.woff | Bin 0 -> 98024 bytes .../v1.13/fonts/fontawesome-webfont.woff2 | Bin 0 -> 77160 bytes .../generated/kubernetes-api/v1.13/index.html | 50930 ++++++++++++++++ .../v1.13/jquery.scrollTo.min.js | 7 + .../generated/kubernetes-api/v1.13/navData.js | 1 + .../generated/kubernetes-api/v1.13/scroll.js | 196 + .../kubeadm-ha-topology-external-etcd.svg | 1 + .../kubeadm-ha-topology-stacked-etcd.svg | 1 + update-imported-docs/reference.yml | 2 +- update-imported-docs/release.yml | 2 +- 347 files changed, 69250 insertions(+), 4447 deletions(-) create mode 100755 content/en/docs/reference/federation/federation/v1beta1/definitions.html create mode 100755 content/en/docs/reference/federation/federation/v1beta1/operations.html create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs.md rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_renew.md => kubeadm_alpha_certs_renew.md} (95%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_renew_all.md => kubeadm_alpha_certs_renew_all.md} (77%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_renew_apiserver-etcd-client.md => kubeadm_alpha_certs_renew_apiserver-etcd-client.md} (78%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_renew_apiserver-kubelet-client.md => kubeadm_alpha_certs_renew_apiserver-kubelet-client.md} (78%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_renew_apiserver.md => kubeadm_alpha_certs_renew_apiserver.md} (73%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_renew_etcd-healthcheck-client.md => kubeadm_alpha_certs_renew_etcd-healthcheck-client.md} (78%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_renew_etcd-peer.md => kubeadm_alpha_certs_renew_etcd-peer.md} (78%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_renew_etcd-server.md => kubeadm_alpha_certs_renew_etcd-server.md} (78%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_renew_front-proxy-client.md => kubeadm_alpha_certs_renew_front-proxy-client.md} (78%) create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubeconfig.md rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_kubeconfig_user.md => kubeadm_alpha_kubeconfig_user.md} (85%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_kubelet.md => kubeadm_alpha_kubelet.md} (95%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_kubelet_config.md => kubeadm_alpha_kubelet_config.md} (96%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_kubelet_config_download.md => kubeadm_alpha_kubelet_config_download.md} (95%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_kubelet_config_enable-dynamic.md => kubeadm_alpha_kubelet_config_enable-dynamic.md} (95%) delete mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_all.md delete mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_cluster-info.md delete mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_create.md delete mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_node_allow-auto-approve.md delete mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_node_allow-post-csrs.md delete mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_all.md delete mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_all.md delete mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config_write-to-disk.md delete mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_write-env-file.md delete mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_preflight.md rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_etcd.md => kubeadm_alpha_preflight.md} (92%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_preflight_node.md => kubeadm_alpha_preflight_node.md} (89%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_selfhosting.md => kubeadm_alpha_selfhosting.md} (100%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_selfhosting_convert-from-staticpods.md => kubeadm_alpha_selfhosting_pivot.md} (76%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_bootstrap-token_node.md => kubeadm_config_print.md} (83%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_config_print-default.md => kubeadm_config_print_init-defaults.md} (61%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_upload-config.md => kubeadm_config_print_join-defaults.md} (59%) create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase.md rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_addon.md => kubeadm_init_phase_addon.md} (95%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_addon_all.md => kubeadm_init_phase_addon_all.md} (70%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_addon_coredns.md => kubeadm_init_phase_addon_coredns.md} (80%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_addon_kube-proxy.md => kubeadm_init_phase_addon_kube-proxy.md} (77%) create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_bootstrap-token.md rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs.md => kubeadm_init_phase_certs.md} (93%) create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_all.md rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_apiserver-etcd-client.md => kubeadm_init_phase_certs_apiserver-etcd-client.md} (71%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_apiserver-kubelet-client.md => kubeadm_init_phase_certs_apiserver-kubelet-client.md} (71%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_apiserver.md => kubeadm_init_phase_certs_apiserver.md} (66%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_ca.md => kubeadm_init_phase_certs_ca.md} (76%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_etcd-ca.md => kubeadm_init_phase_certs_etcd-ca.md} (88%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_etcd-healthcheck-client.md => kubeadm_init_phase_certs_etcd-healthcheck-client.md} (71%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_etcd-peer.md => kubeadm_init_phase_certs_etcd-peer.md} (72%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_etcd-server.md => kubeadm_init_phase_certs_etcd-server.md} (69%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_front-proxy-ca.md => kubeadm_init_phase_certs_front-proxy-ca.md} (88%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_front-proxy-client.md => kubeadm_init_phase_certs_front-proxy-client.md} (70%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_certs_sa.md => kubeadm_init_phase_certs_sa.md} (68%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_controlplane.md => kubeadm_init_phase_control-plane.md} (91%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_controlplane_all.md => kubeadm_init_phase_control-plane_all.md} (73%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_controlplane_apiserver.md => kubeadm_init_phase_control-plane_apiserver.md} (74%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_controlplane_controller-manager.md => kubeadm_init_phase_control-plane_controller-manager.md} (71%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_controlplane_scheduler.md => kubeadm_init_phase_control-plane_scheduler.md} (74%) create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_etcd.md rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_etcd_local.md => kubeadm_init_phase_etcd_local.md} (67%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_kubeconfig.md => kubeadm_init_phase_kubeconfig.md} (95%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_kubeconfig_admin.md => kubeadm_init_phase_kubeconfig_admin.md} (82%) create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_all.md rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_kubeconfig_controller-manager.md => kubeadm_init_phase_kubeconfig_controller-manager.md} (81%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_kubeconfig_kubelet.md => kubeadm_init_phase_kubeconfig_kubelet.md} (73%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_kubeconfig_scheduler.md => kubeadm_init_phase_kubeconfig_scheduler.md} (82%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_mark-master.md => kubeadm_init_phase_kubelet-start.md} (58%) create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_mark-control-plane.md rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_preflight_master.md => kubeadm_init_phase_preflight.md} (81%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase.md => kubeadm_init_phase_upload-config.md} (85%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_bootstrap-token.md => kubeadm_init_phase_upload-config_all.md} (70%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_kubelet_config_annotate-cri.md => kubeadm_init_phase_upload-config_kubeadm.md} (66%) rename content/en/docs/reference/setup-tools/kubeadm/generated/{kubeadm_alpha_phase_kubelet_config_upload.md => kubeadm_init_phase_upload-config_kubelet.md} (76%) create mode 100644 content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md delete mode 100644 content/en/docs/reference/setup-tools/kubefed/kubefed-init.md delete mode 100644 content/en/docs/reference/setup-tools/kubefed/kubefed-join.md delete mode 100644 content/en/docs/reference/setup-tools/kubefed/kubefed-options.md delete mode 100644 content/en/docs/reference/setup-tools/kubefed/kubefed-unjoin.md delete mode 100644 content/en/docs/reference/setup-tools/kubefed/kubefed-version.md delete mode 100644 content/en/docs/setup/custom-cloud/master.yaml delete mode 100644 content/en/docs/setup/custom-cloud/node.yaml create mode 100644 content/en/docs/setup/independent/ha-topology.md create mode 100644 content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md create mode 100644 content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13.md create mode 100644 content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha-1-13.md create mode 100644 content/ko/case-studies/adform/adform_featured_logo.png create mode 100644 content/ko/case-studies/adform/index.html create mode 100644 content/ko/case-studies/amadeus/amadeus_featured.png create mode 100644 content/ko/case-studies/amadeus/amadeus_logo.png create mode 100644 content/ko/case-studies/amadeus/index.html create mode 100644 content/ko/case-studies/ancestry/ancestry_featured.png create mode 100644 content/ko/case-studies/ancestry/ancestry_logo.png create mode 100644 content/ko/case-studies/ancestry/index.html create mode 100644 content/ko/case-studies/blablacar/blablacar_featured.png create mode 100644 content/ko/case-studies/blablacar/blablacar_logo.png create mode 100644 content/ko/case-studies/blablacar/index.html create mode 100644 content/ko/case-studies/blackrock/blackrock_featured.png create mode 100644 content/ko/case-studies/blackrock/blackrock_logo.png create mode 100644 content/ko/case-studies/blackrock/index.html create mode 100644 content/ko/case-studies/box/box_featured.png create mode 100644 content/ko/case-studies/box/box_logo.png create mode 100644 content/ko/case-studies/box/box_small.png create mode 100644 content/ko/case-studies/box/index.html create mode 100644 content/ko/case-studies/box/video.png create mode 100644 content/ko/case-studies/buffer/buffer_featured.png create mode 100644 content/ko/case-studies/buffer/buffer_logo.png create mode 100644 content/ko/case-studies/buffer/index.html create mode 100644 content/ko/case-studies/capital-one/capitalone_featured_logo.png create mode 100644 content/ko/case-studies/capital-one/index.html create mode 100644 content/ko/case-studies/ccp-games/ccp_logo.png create mode 100644 content/ko/case-studies/ccp-games/index.html create mode 100644 content/ko/case-studies/comcast/comcast_logo.png create mode 100644 content/ko/case-studies/comcast/index.html create mode 100644 content/ko/case-studies/concur/concur_featured_logo.png create mode 100644 content/ko/case-studies/concur/index.html create mode 100644 content/ko/case-studies/crowdfire/crowdfire_featured_logo.png create mode 100644 content/ko/case-studies/crowdfire/index.html create mode 100644 content/ko/case-studies/ebay/ebay_featured.png create mode 100644 content/ko/case-studies/ebay/ebay_logo.png create mode 100644 content/ko/case-studies/ebay/index.html create mode 100644 content/ko/case-studies/goldman-sachs/gs_logo.png create mode 100644 content/ko/case-studies/goldman-sachs/index.html create mode 100644 content/ko/case-studies/golfnow/golfnow_featured.png create mode 100644 content/ko/case-studies/golfnow/golfnow_logo.png create mode 100644 content/ko/case-studies/golfnow/index.html create mode 100644 content/ko/case-studies/haufegroup/haufegroup_featured.png create mode 100644 content/ko/case-studies/haufegroup/haufegroup_logo.png create mode 100644 content/ko/case-studies/haufegroup/index.html create mode 100644 content/ko/case-studies/homeoffice/homeoffice_logo.png create mode 100644 content/ko/case-studies/homeoffice/index.html create mode 100644 content/ko/case-studies/huawei/huawei_featured.png create mode 100644 content/ko/case-studies/huawei/huawei_logo.png create mode 100644 content/ko/case-studies/huawei/index.html create mode 100644 content/ko/case-studies/ibm/ibm_featured_logo.png create mode 100644 content/ko/case-studies/ibm/ibm_featured_logo.svg create mode 100644 content/ko/case-studies/ibm/index.html create mode 100644 content/ko/case-studies/ing/index.html create mode 100644 content/ko/case-studies/ing/ing_featured_logo.png create mode 100644 content/ko/case-studies/jd/index.html create mode 100644 content/ko/case-studies/jd/jd_logo.png create mode 100644 content/ko/case-studies/liveperson/index.html create mode 100644 content/ko/case-studies/liveperson/liveperson_logo.png create mode 100644 content/ko/case-studies/monzo/index.html create mode 100644 content/ko/case-studies/monzo/monzo_logo.png create mode 100644 content/ko/case-studies/naic/index.html create mode 100644 content/ko/case-studies/naic/naic_featured_logo.png create mode 100644 content/ko/case-studies/newyorktimes/index.html create mode 100644 content/ko/case-studies/newyorktimes/newyorktimes_featured.png create mode 100644 content/ko/case-studies/newyorktimes/newyorktimes_logo.png create mode 100644 content/ko/case-studies/nordstrom/index.html create mode 100644 content/ko/case-studies/nordstrom/nordstrom_featured_logo.png create mode 100644 content/ko/case-studies/northwestern-mutual/index.html create mode 100644 content/ko/case-studies/northwestern-mutual/northwestern_featured_logo.png create mode 100644 content/ko/case-studies/ocado/index.html create mode 100644 content/ko/case-studies/ocado/ocado_featured_logo.png create mode 100644 content/ko/case-studies/openAI/index.html create mode 100644 content/ko/case-studies/openAI/openai_featured.png create mode 100644 content/ko/case-studies/openAI/openai_logo.png create mode 100644 content/ko/case-studies/peardeck/index.html create mode 100644 content/ko/case-studies/peardeck/peardeck_featured.png create mode 100644 content/ko/case-studies/peardeck/peardeck_logo.png create mode 100644 content/ko/case-studies/pearson/index.html create mode 100644 content/ko/case-studies/pearson/pearson_featured.png create mode 100644 content/ko/case-studies/pearson/pearson_logo.png create mode 100644 content/ko/case-studies/philips/index.html create mode 100644 content/ko/case-studies/philips/philips_logo.png create mode 100644 content/ko/case-studies/pinterest/index.html create mode 100644 content/ko/case-studies/pinterest/pinterest_feature.png create mode 100644 content/ko/case-studies/pinterest/pinterest_logo.png create mode 100644 content/ko/case-studies/pokemon-go/index.html create mode 100644 content/ko/case-studies/pokemon-go/pokemon_go_logo.png create mode 100644 content/ko/case-studies/samsung-sds/index.html create mode 100644 content/ko/case-studies/samsung-sds/sds_logo.png create mode 100644 content/ko/case-studies/sap/index.html create mode 100644 content/ko/case-studies/sap/sap_logo.png create mode 100644 content/ko/case-studies/sap/sap_small.png create mode 100644 content/ko/case-studies/slingtv/index.html create mode 100644 content/ko/case-studies/slingtv/slingtv_featured_logo.png create mode 100644 content/ko/case-studies/soundcloud/index.html create mode 100644 content/ko/case-studies/soundcloud/soundcloud_logo.png create mode 100644 content/ko/case-studies/squarespace/index.html create mode 100644 content/ko/case-studies/squarespace/squarespace_featured_logo.png create mode 100644 content/ko/case-studies/wepay/index.html create mode 100644 content/ko/case-studies/wepay/wepay_logo.png create mode 100644 content/ko/case-studies/wikimedia/index.html create mode 100644 content/ko/case-studies/wikimedia/wikimedia_featured.png create mode 100644 content/ko/case-studies/wikimedia/wikimedia_logo.png create mode 100644 content/ko/case-studies/wink/index.html create mode 100644 content/ko/case-studies/wink/wink_featured.png create mode 100644 content/ko/case-studies/wink/wink_logo.png create mode 100644 content/ko/case-studies/workiva/index.html create mode 100644 content/ko/case-studies/workiva/workiva_featured_logo.png create mode 100644 content/ko/case-studies/yahoo-japan/index.html create mode 100644 content/ko/case-studies/yahoo-japan/yahooJapan_logo.png create mode 100644 content/ko/case-studies/ygrene/index.html create mode 100644 content/ko/case-studies/ygrene/ygrene_featured_logo.png create mode 100644 content/ko/case-studies/zalando/index.html create mode 100644 content/ko/case-studies/zalando/zalando_feature_logo.png create mode 100644 content/ko/case-studies/zulily/index.html create mode 100644 content/ko/case-studies/zulily/zulily_featured.png create mode 100644 content/ko/case-studies/zulily/zulily_logo.png create mode 100644 content/ko/docs/_index.md create mode 100644 content/ko/docs/concepts/overview/components.md create mode 100644 content/ko/docs/concepts/overview/kubernetes-api.md create mode 100644 content/ko/docs/concepts/overview/working-with-objects/_index.md create mode 100644 content/ko/docs/concepts/overview/working-with-objects/kubernetes-objects.md create mode 100644 content/ko/docs/concepts/workloads/_index.md create mode 100644 content/ko/docs/concepts/workloads/pods/_index.md create mode 100644 content/ko/docs/concepts/workloads/pods/pod-overview.md create mode 100644 content/ko/docs/reference/_index.md create mode 100644 content/ko/docs/reference/glossary/controller.md create mode 100644 content/ko/docs/reference/glossary/etcd.md create mode 100644 content/ko/docs/reference/glossary/kube-apiserver.md create mode 100644 content/ko/docs/reference/glossary/kube-controller-manager.md create mode 100644 content/ko/docs/reference/glossary/kube-scheduler.md create mode 100644 content/ko/docs/reference/glossary/kubelet.md create mode 100644 content/ko/docs/tasks/_index.md create mode 100644 content/ko/docs/tasks/run-application/_index.md create mode 100644 content/ko/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md create mode 100644 content/ko/examples/application/deployment.yaml create mode 100644 content/ko/examples/application/hpa/php-apache.yaml create mode 100644 content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md create mode 100644 content/zh/docs/reference/setup-tools/kubeadm/kubeadm-init.md create mode 100644 content/zh/docs/reference/setup-tools/kubeadm/kubeadm.md create mode 100644 content/zh/docs/setup/independent/create-cluster-kubeadm.md create mode 100644 content/zh/docs/setup/independent/install-kubeadm.md create mode 100644 static/docs/reference/generated/kubernetes-api/v1.13/css/bootstrap.min.css create mode 100644 static/docs/reference/generated/kubernetes-api/v1.13/css/font-awesome.min.css create mode 100644 static/docs/reference/generated/kubernetes-api/v1.13/css/stylesheet.css create mode 100644 static/docs/reference/generated/kubernetes-api/v1.13/fonts/FontAwesome.otf create mode 100644 static/docs/reference/generated/kubernetes-api/v1.13/fonts/fontawesome-webfont.eot create mode 100644 static/docs/reference/generated/kubernetes-api/v1.13/fonts/fontawesome-webfont.svg create mode 100644 static/docs/reference/generated/kubernetes-api/v1.13/fonts/fontawesome-webfont.ttf create mode 100644 static/docs/reference/generated/kubernetes-api/v1.13/fonts/fontawesome-webfont.woff create mode 100644 static/docs/reference/generated/kubernetes-api/v1.13/fonts/fontawesome-webfont.woff2 create mode 100644 static/docs/reference/generated/kubernetes-api/v1.13/index.html create mode 100644 static/docs/reference/generated/kubernetes-api/v1.13/jquery.scrollTo.min.js create mode 100644 static/docs/reference/generated/kubernetes-api/v1.13/navData.js create mode 100644 static/docs/reference/generated/kubernetes-api/v1.13/scroll.js create mode 100755 static/images/kubeadm/kubeadm-ha-topology-external-etcd.svg create mode 100755 static/images/kubeadm/kubeadm-ha-topology-stacked-etcd.svg diff --git a/config.toml b/config.toml index 5b6f888bc7..65bdd3192c 100644 --- a/config.toml +++ b/config.toml @@ -63,10 +63,10 @@ time_format_blog = "Monday, January 02, 2006" description = "Production-Grade Container Orchestration" showedit = true -latest = "v1.12" +latest = "v1.13" -fullversion = "v1.12.0" -version = "v1.12" +fullversion = "v1.13.0" +version = "v1.13" githubbranch = "master" docsbranch = "master" deprecated = false @@ -76,10 +76,10 @@ githubWebsiteRepo = "github.com/kubernetes/website" githubWebsiteRaw = "raw.githubusercontent.com/kubernetes/website" [[params.versions]] -fullversion = "v1.12.0" -version = "v1.12" -githubbranch = "v1.12.0" -docsbranch = "release-1.12" +fullversion = "v1.13.0" +version = "v1.13" +githubbranch = "v1.13.0" +docsbranch = "release-1.13" url = "https://kubernetes.io" [params.pushAssets] @@ -94,34 +94,33 @@ js = [ ] [[params.versions]] -fullversion = "v1.11.3" +fullversion = "v1.12.3" +version = "v1.12" +githubbranch = "v1.12.3" +docsbranch = "release-1.12" +url = "https://v1-12.docs.kubernetes.io" + +[[params.versions]] +fullversion = "v1.11.5" version = "v1.11" -githubbranch = "v1.11.3" +githubbranch = "v1.11.5" docsbranch = "release-1.11" url = "https://v1-11.docs.kubernetes.io" [[params.versions]] -fullversion = "v1.10.3" +fullversion = "v1.10.11" version = "v1.10" -githubbranch = "v1.10.3" +githubbranch = "v1.10.11" docsbranch = "release-1.10" url = "https://v1-10.docs.kubernetes.io" [[params.versions]] -fullversion = "v1.9.7" +fullversion = "v1.9.11" version = "v1.9" -githubbranch = "v1.9.7" +githubbranch = "v1.9.11" docsbranch = "release-1.9" url = "https://v1-9.docs.kubernetes.io" -[[params.versions]] -fullversion = "v1.8.4" -version = "v1.8" -githubbranch = "v1.8.4" -docsbranch = "release-1.8" -url = "https://v1-8.docs.kubernetes.io" - - # Language definitions. [languages] diff --git a/content/en/docs/concepts/architecture/nodes.md b/content/en/docs/concepts/architecture/nodes.md index 5a4d84dc17..f82c271345 100644 --- a/content/en/docs/concepts/architecture/nodes.md +++ b/content/en/docs/concepts/architecture/nodes.md @@ -158,6 +158,20 @@ to be unreachable. (The default timeouts are 40s to start reporting ConditionUnknown and 5m after that to start evicting pods.) The node controller checks the state of each node every `--node-monitor-period` seconds. +In versions of Kubernetes prior to 1.13, NodeStatus is the heartbeat from the +node. Starting from Kubernetes 1.13, node lease feature is introduced as an +alpha feature (feature gate `NodeLease`, +[KEP-0009](https://github.com/kubernetes/community/blob/master/keps/sig-node/0009-node-heartbeat.md)). +When node lease feature is enabled, each node has an associated `Lease` object in +`kube-node-lease` namespace that is renewed by the node periodically, and both +NodeStatus and node lease are treated as heartbeats from the node. Node leases +are renewed frequently while NodeStatus is reported from node to master only +when there is some change or enough time has passed (default is 1 minute, which +is longer than the default timeout of 40 seconds for unreachable nodes). Since +node lease is much more lightweight than NodeStatus, this feature makes node +heartbeat significantly cheaper from both scalability and performance +perspectives. + In Kubernetes 1.4, we updated the logic of the node controller to better handle cases when a large number of nodes have problems with reaching the master (e.g. because the master has networking problem). Starting with 1.4, the node @@ -212,11 +226,12 @@ For self-registration, the kubelet is started with the following options: - `--register-node` - Automatically register with the API server. - `--register-with-taints` - Register the node with the given list of taints (comma separated `=:`). No-op if `register-node` is false. - `--node-ip` - IP address of the node. - - `--node-labels` - Labels to add when registering the node in the cluster. + - `--node-labels` - Labels to add when registering the node in the cluster (see label restrictions enforced by the [NodeRestriction admission plugin](/docs/reference/access-authn-authz/admission-controllers/#noderestriction) in 1.13+). - `--node-status-update-frequency` - Specifies how often kubelet posts node status to master. -Currently, any kubelet is authorized to create/modify any node resource, but in practice it only creates/modifies -its own. (In the future, we plan to only allow a kubelet to modify its own node resource.) +When the [Node authorization mode](/docs/reference/access-authn-authz/node/) and +[NodeRestriction admission plugin](/docs/reference/access-authn-authz/admission-controllers/#noderestriction) are enabled, +kubelets are only authorized to create/modify their own Node resource. #### Manual Node Administration diff --git a/content/en/docs/concepts/cluster-administration/cloud-providers.md b/content/en/docs/concepts/cluster-administration/cloud-providers.md index 7fba2d0fb0..4f868a62ca 100644 --- a/content/en/docs/concepts/cluster-administration/cloud-providers.md +++ b/content/en/docs/concepts/cluster-administration/cloud-providers.md @@ -17,30 +17,32 @@ kubeadm has configuration options to specify configuration information for cloud in-tree cloud provider can be configured using kubeadm as shown below: ```yaml -apiVersion: kubeadm.k8s.io/v1alpha3 +apiVersion: kubeadm.k8s.io/v1beta1 kind: InitConfiguration nodeRegistration: kubeletExtraArgs: cloud-provider: "openstack" cloud-config: "/etc/kubernetes/cloud.conf" --- +apiVersion: kubeadm.k8s.io/v1beta1 kind: ClusterConfiguration -apiVersion: kubeadm.k8s.io/v1alpha3 -kubernetesVersion: v1.12.0 -apiServerExtraArgs: - cloud-provider: "openstack" - cloud-config: "/etc/kubernetes/cloud.conf" -apiServerExtraVolumes: -- name: cloud - hostPath: "/etc/kubernetes/cloud.conf" - mountPath: "/etc/kubernetes/cloud.conf" -controllerManagerExtraArgs: - cloud-provider: "openstack" - cloud-config: "/etc/kubernetes/cloud.conf" -controllerManagerExtraVolumes: -- name: cloud - hostPath: "/etc/kubernetes/cloud.conf" - mountPath: "/etc/kubernetes/cloud.conf" +kubernetesVersion: v1.13.0 +apiServer: + extraArgs: + cloud-provider: "openstack" + cloud-config: "/etc/kubernetes/cloud.conf" + extraVolumes: + - name: cloud + hostPath: "/etc/kubernetes/cloud.conf" + mountPath: "/etc/kubernetes/cloud.conf" +controllerManager: + extraArgs: + cloud-provider: "openstack" + cloud-config: "/etc/kubernetes/cloud.conf" + extraVolumes: + - name: cloud + hostPath: "/etc/kubernetes/cloud.conf" + mountPath: "/etc/kubernetes/cloud.conf" ``` The in-tree cloud providers typically need both `--cloud-provider` and `--cloud-config` specified in the command lines diff --git a/content/en/docs/concepts/configuration/assign-pod-node.md b/content/en/docs/concepts/configuration/assign-pod-node.md index 5a0f861654..468ca73303 100644 --- a/content/en/docs/concepts/configuration/assign-pod-node.md +++ b/content/en/docs/concepts/configuration/assign-pod-node.md @@ -92,6 +92,21 @@ For example, the value of `kubernetes.io/hostname` may be the same as the Node n and a different value in other environments. {{< /note >}} +## Node isolation/restriction + +Adding labels to Node objects allows targeting pods to specific nodes or groups of nodes. +This can be used to ensure specific pods only run on nodes with certain isolation, security, or regulatory properties. +When using labels for this purpose, choosing label keys that cannot be modified by the kubelet process on the node is strongly recommended. +This prevents a compromised node from using its kubelet credential to set those labels on its own Node object, +and influencing the scheduler to schedule workloads to the compromised node. + +The `NodeRestriction` admission plugin prevents kubelets from setting or modifying labels with a `node-restriction.kubernetes.io/` prefix. +To make use of that label prefix for node isolation: + +1. Ensure you are using the [Node authorizer](/docs/reference/access-authn-authz/node/) and have enabled the [NodeRestriction admission plugin](/docs/reference/access-authn-authz/admission-controllers/#noderestriction). +2. Add labels under the `node-restriction.kubernetes.io/` prefix to your Node objects, and use those labels in your node selectors. +For example, `example.com.node-restriction.kubernetes.io/fips=true` or `example.com.node-restriction.kubernetes.io/pci-dss=true`. + ## Affinity and anti-affinity `nodeSelector` provides a very simple way to constrain pods to nodes with particular labels. The affinity/anti-affinity diff --git a/content/en/docs/concepts/configuration/taint-and-toleration.md b/content/en/docs/concepts/configuration/taint-and-toleration.md index 074cd82c02..a66b4f4a43 100644 --- a/content/en/docs/concepts/configuration/taint-and-toleration.md +++ b/content/en/docs/concepts/configuration/taint-and-toleration.md @@ -223,9 +223,7 @@ certain condition is true. The following taints are built in: as unusable. After a controller from the cloud-controller-manager initializes this node, the kubelet removes this taint. -When the `TaintBasedEvictions` alpha feature is enabled (you can do this by -including `TaintBasedEvictions=true` in `--feature-gates` for Kubernetes controller manager, -such as `--feature-gates=FooBar=true,TaintBasedEvictions=true`), the taints are automatically +In version 1.13, the `TaintBasedEvictions` feature is promoted to beta and enabled by default, hence the taints are automatically added by the NodeController (or kubelet) and the normal logic for evicting pods from nodes based on the Ready NodeCondition is disabled. @@ -236,7 +234,7 @@ in a rate-limited way. This prevents massive pod evictions in scenarios such as the master becoming partitioned from the nodes. {{< /note >}} -This alpha feature, in combination with `tolerationSeconds`, allows a pod +This beta feature, in combination with `tolerationSeconds`, allows a pod to specify how long it should stay bound to a node that has one or both of these problems. For example, an application with a lot of local state might want to stay @@ -246,7 +244,7 @@ The toleration the pod would use in that case would look like ```yaml tolerations: -- key: "node.alpha.kubernetes.io/unreachable" +- key: "node.kubernetes.io/unreachable" operator: "Exists" effect: "NoExecute" tolerationSeconds: 6000 @@ -257,9 +255,9 @@ Note that Kubernetes automatically adds a toleration for unless the pod configuration provided by the user already has a toleration for `node.kubernetes.io/not-ready`. Likewise it adds a toleration for -`node.alpha.kubernetes.io/unreachable` with `tolerationSeconds=300` +`node.kubernetes.io/unreachable` with `tolerationSeconds=300` unless the pod configuration provided -by the user already has a toleration for `node.alpha.kubernetes.io/unreachable`. +by the user already has a toleration for `node.kubernetes.io/unreachable`. These automatically-added tolerations ensure that the default pod behavior of remaining bound for 5 minutes after one of these @@ -270,7 +268,7 @@ admission controller](https://git.k8s.io/kubernetes/plugin/pkg/admission/default [DaemonSet](/docs/concepts/workloads/controllers/daemonset/) pods are created with `NoExecute` tolerations for the following taints with no `tolerationSeconds`: - * `node.alpha.kubernetes.io/unreachable` + * `node.kubernetes.io/unreachable` * `node.kubernetes.io/not-ready` This ensures that DaemonSet pods are never evicted due to these problems, diff --git a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md index fe31589d1c..cf9c8fdf57 100644 --- a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md +++ b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md @@ -136,6 +136,36 @@ a Kubernetes release with a newer device plugin API version, upgrade your device to support both versions before upgrading these nodes to ensure the continuous functioning of the device allocations during the upgrade. +## Monitoring Device Plugin Resources + +In order to monitor resources provided by device plugins, monitoring agents need to be able to +discover the set of devices that are in-use on the node and obtain metadata to describe which +container the metric should be associated with. Prometheus metrics exposed by device monitoring +agents should follow the +[Kubernetes Instrumentation Guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/instrumentation.md), +which requires identifying containers using `pod`, `namespace`, and `container` prometheus labels. +The kubelet provides a gRPC service to enable discovery of in-use devices, and to provide metadata +for these devices: + +```gRPC +// PodResources is a service provided by the kubelet that provides information about the +// node resources consumed by pods and containers on the node +service PodResources { + rpc List(ListPodResourcesRequest) returns (ListPodResourcesResponse) {} +} +``` + +The gRPC service is served over a unix socket at `/var/lib/kubelet/pod-resources/kubelet.sock`. +Monitoring agents for device plugin resources can be deployed as a daemon, or as a DaemonSet. +The cannonical directory `/var/lib/kubelet/pod-resources` requires privileged access, so monitoring +agents must run in a privileged security context. If a device monitoring agent is running as a +DaemonSet, `/var/lib/kubelet/pod-resources` must be mounted as a +[Volume](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#volume-v1-core) +in the plugin's +[PodSpec](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podspec-v1-core). + +Support for the "PodResources service" is still in alpha. + ## Examples For examples of device plugin implementations, see: diff --git a/content/en/docs/concepts/overview/object-management-kubectl/declarative-config.md b/content/en/docs/concepts/overview/object-management-kubectl/declarative-config.md index 48025a12e1..70fc566dfc 100644 --- a/content/en/docs/concepts/overview/object-management-kubectl/declarative-config.md +++ b/content/en/docs/concepts/overview/object-management-kubectl/declarative-config.md @@ -9,7 +9,8 @@ Kubernetes objects can be created, updated, and deleted by storing multiple object configuration files in a directory and using `kubectl apply` to recursively create and update those objects as needed. This method retains writes made to live objects without merging the changes -back into the object configuration files. +back into the object configuration files. `kubectl diff` also gives you a +preview of what changes `apply` will make. {{% /capture %}} {{% capture body %}} @@ -67,6 +68,14 @@ Here's an example of an object configuration file: {{< codenew file="application/simple_deployment.yaml" >}} +Run `kubectl diff` to print the object that will be created: +```shell +kubectl diff -f https://k8s.io/examples/application/simple_deployment.yaml +``` +{{< note >}} +**Note:** `diff` uses [server-side dry-run](/docs/reference/using-api/api-concepts/#dry-run), which needs to be enabled on `kube-apiserver`. +{{< /note >}} + Create the object using `kubectl apply`: ```shell @@ -130,6 +139,7 @@ if those objects already exist. This approach accomplishes the following: 2. Clears fields removed from the configuration file in the live configuration. ```shell +kubectl diff -f / kubectl apply -f / ``` @@ -262,6 +272,7 @@ Update the `simple_deployment.yaml` configuration file to change the image from Apply the changes made to the configuration file: ```shell +kubectl diff -f https://k8s.io/examples/application/update_deployment.yaml kubectl apply -f https://k8s.io/examples/application/update_deployment.yaml ``` @@ -977,5 +988,3 @@ template: - [Kubectl Command Reference](/docs/reference/generated/kubectl/kubectl/) - [Kubernetes API Reference](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/) {{% /capture %}} - - diff --git a/content/en/docs/concepts/overview/object-management-kubectl/overview.md b/content/en/docs/concepts/overview/object-management-kubectl/overview.md index 19c8a885f2..3987da4a71 100644 --- a/content/en/docs/concepts/overview/object-management-kubectl/overview.md +++ b/content/en/docs/concepts/overview/object-management-kubectl/overview.md @@ -144,16 +144,19 @@ API operation to replace the entire object configuration. ### Examples -Process all object configuration files in the `configs` directory, and -create or patch the live objects: +Process all object configuration files in the `configs` directory, and create or +patch the live objects. You can first `diff` to see what changes are going to be +made, and then apply: ```sh +kubectl diff -f configs/ kubectl apply -f configs/ ``` Recursively process directories: ```sh +kubectl diff -R -f configs/ kubectl apply -R -f configs/ ``` @@ -181,5 +184,3 @@ Disadvantages compared to imperative object configuration: {{< comment >}} {{< /comment >}} {{% /capture %}} - - diff --git a/content/en/docs/concepts/storage/persistent-volumes.md b/content/en/docs/concepts/storage/persistent-volumes.md index 995881711b..f9f4df2615 100644 --- a/content/en/docs/concepts/storage/persistent-volumes.md +++ b/content/en/docs/concepts/storage/persistent-volumes.md @@ -192,6 +192,7 @@ the following types of volumes: * Azure File * Azure Disk * Portworx +* FlexVolumes You can only expand a PVC if its storage class's `allowVolumeExpansion` field is set to true. @@ -227,16 +228,25 @@ kubectl describe pvc If the `PersistentVolumeClaim` has the status `FileSystemResizePending`, it is safe to recreate the pod using the PersistentVolumeClaim. -#### Resizing an in-use PersistentVolumeClaim +FlexVolumes allow resize if the driver is set with the `RequiresFSResize` capability to true. +The FlexVolume can be resized on pod restart. {{< feature-state for_k8s_version="v1.11" state="alpha" >}} +#### Resizing an in-use PersistentVolumeClaim + Expanding in-use PVCs is an alpha feature. To use it, enable the `ExpandInUsePersistentVolumes` feature gate. In this case, you don't need to delete and recreate a Pod or deployment that is using an existing PVC. Any in-use PVC automatically becomes available to its Pod as soon as its file system has been expanded. This feature has no effect on PVCs that are not in use by a Pod or deployment. You must create a Pod which uses the PVC before the expansion can complete. +Expanding in-use PVCs for FlexVolumes is added in release 1.13. To enable this feature use `ExpandInUsePersistentVolumes` and `ExpandPersistentVolumes` feature gates. The `ExpandPersistentVolumes` feature gate is already enabled by default. If the `ExpandInUsePersistentVolumes` is set, FlexVolume can be resized online without pod restart. + +{{< note >}} +**Note:** FlexVolume resize is possible only when the underlying driver supports resize. +{{< /note >}} + {{< note >}} Expanding EBS volumes is a time consuming operation. Also, there is a per-volume quota of one modification every 6 hours. {{< /note >}} @@ -553,6 +563,7 @@ applicable. * iSCSI * Local volume * RBD (Ceph Block Device) +* VsphereVolume (alpha) {{< note >}} Only FC and iSCSI volumes supported raw block volumes in Kubernetes 1.9. diff --git a/content/en/docs/concepts/storage/storage-classes.md b/content/en/docs/concepts/storage/storage-classes.md index 3aba1eee67..798c409db4 100644 --- a/content/en/docs/concepts/storage/storage-classes.md +++ b/content/en/docs/concepts/storage/storage-classes.md @@ -121,13 +121,6 @@ the class or PV, so mount of the PV will simply fail if one is invalid. ### Volume Binding Mode -{{< feature-state for_k8s_version="v1.12" state="beta" >}} - -{{< note >}} -This feature requires the `VolumeScheduling` feature gate to be -enabled. -{{< /note >}} - The `volumeBindingMode` field controls when [volume binding and dynamic provisioning](/docs/concepts/storage/persistent-volumes/#provisioning) should occur. @@ -159,12 +152,6 @@ The following plugins support `WaitForFirstConsumer` with pre-created Persistent * [Local](#local) ### Allowed Topologies -{{< feature-state for_k8s_version="v1.12" state="beta" >}} - -{{< note >}} -This feature requires the `VolumeScheduling` feature gate to be -enabled. -{{< /note >}} When a cluster operactor specifies the `WaitForFirstConsumer` volume binding mode, it is no longer necessary to restrict provisioning to specific topologies in most situations. However, diff --git a/content/en/docs/concepts/storage/volumes.md b/content/en/docs/concepts/storage/volumes.md index 3820394deb..4c35d92678 100644 --- a/content/en/docs/concepts/storage/volumes.md +++ b/content/en/docs/concepts/storage/volumes.md @@ -1139,16 +1139,25 @@ to [this FAQ](https://github.com/kubernetes/community/blob/master/sig-storage/vo ### CSI -{{< feature-state for_k8s_version="v1.10" state="beta" >}} - [Container Storage Interface](https://github.com/container-storage-interface/spec/blob/master/spec.md) (CSI) defines a standard interface for container orchestration systems (like Kubernetes) to expose arbitrary storage systems to their container workloads. Please read the [CSI design proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md) for more information. -CSI support was introduced as alpha in Kubernetes v1.9 and moved to beta in -Kubernetes v1.10. +CSI support was introduced as alpha in Kubernetes v1.9, moved to beta in +Kubernetes v1.10, and is GA in Kubernetes v1.13. + +{{< note >}} +**Note:** Support for CSI spec versions 0.2 and 0.3 are deprecated in Kubernetes +v1.13 and will be removed in a future release. +{{< /note >}} + +{{< note >}} +**Note:** CSI drivers may not be compatible across all Kubernetes releases. +Please check the specific CSI driver's documentation for supported +deployments steps for each Kubernetes release and a compatibility matrix. +{{< /note >}} Once a CSI compatible volume driver is deployed on a Kubernetes cluster, users may use the `csi` volume type to attach, mount, etc. the volumes exposed by the @@ -1224,6 +1233,10 @@ Kubernetes component using the following feature gate flags: Learn how to [setup your PV/PVC with raw block volume support](/docs/concepts/storage/persistent-volumes/#raw-block-volume-support). +#### Developer resources +For more information on how to develop a CSI driver, refer to the [kubernetes-csi +documentation](https://kubernetes-csi.github.io/docs/) + ### Flexvolume Flexvolume is an out-of-tree plugin interface that has existed in Kubernetes diff --git a/content/en/docs/concepts/workloads/controllers/daemonset.md b/content/en/docs/concepts/workloads/controllers/daemonset.md index 5fd9fd4396..3a205a2f79 100644 --- a/content/en/docs/concepts/workloads/controllers/daemonset.md +++ b/content/en/docs/concepts/workloads/controllers/daemonset.md @@ -161,14 +161,14 @@ Although Daemon Pods respect the following tolerations are added to DaemonSet Pods automatically according to the related features. -| Toleration Key | Effect | Alpha Features | Version | Description | -| ---------------------------------------- | ---------- | ------------------------------------------------------------ | ------- | ------------------------------------------------------------ | -| `node.kubernetes.io/not-ready` | NoExecute | `TaintBasedEvictions` | 1.8+ | When `TaintBasedEvictions` is enabled, they will not be evicted when there are node problems such as a network partition. | -| `node.kubernetes.io/unreachable` | NoExecute | `TaintBasedEvictions` | 1.8+ | When `TaintBasedEvictions` is enabled, they will not be evicted when there are node problems such as a network partition. | -| `node.kubernetes.io/disk-pressure` | NoSchedule | | 1.8+ | | -| `node.kubernetes.io/memory-pressure` | NoSchedule | | 1.8+ | | -| `node.kubernetes.io/unschedulable` | NoSchedule | | 1.12+ | DaemonSet pods tolerate unschedulable attributes by default scheduler. | -| `node.kubernetes.io/network-unavailable` | NoSchedule | | 1.12+ | DaemonSet pods, who uses host network, tolerate network-unavailable attributes by default scheduler. | +| Toleration Key | Effect | Version | Description | +| ---------------------------------------- | ---------- | ------- | ------------------------------------------------------------ | +| `node.kubernetes.io/not-ready` | NoExecute | 1.13+ | DaemonSet pods will not be evicted when there are node problems such as a network partition. | +| `node.kubernetes.io/unreachable` | NoExecute | 1.13+ | DaemonSet pods will not be evicted when there are node problems such as a network partition. | +| `node.kubernetes.io/disk-pressure` | NoSchedule | 1.8+ | | +| `node.kubernetes.io/memory-pressure` | NoSchedule | 1.8+ | | +| `node.kubernetes.io/unschedulable` | NoSchedule | 1.12+ | DaemonSet pods tolerate unschedulable attributes by default scheduler. | +| `node.kubernetes.io/network-unavailable` | NoSchedule | 1.12+ | DaemonSet pods, who uses host network, tolerate network-unavailable attributes by default scheduler. | diff --git a/content/en/docs/getting-started-guides/windows/_index.md b/content/en/docs/getting-started-guides/windows/_index.md index 5243a6ccfb..71107fe827 100644 --- a/content/en/docs/getting-started-guides/windows/_index.md +++ b/content/en/docs/getting-started-guides/windows/_index.md @@ -156,6 +156,33 @@ Note: this file assumes that a user previous created 'l2bridge' host networks on } ``` +#### DNS configurations + +DNS configurations for Windows containers are set by CNI plugins which support `dns` capabilities. To enable `dns` capabilities, the following options should be included in the CNI configuration file: + +```json +{ + ... + "capabilities": {"dns": true}, +} +``` + +The following DNS options from kubelet will be passed to CNI plugins: + +- servers: List of DNS servers. +- searches: List of DNS search domains. +- options: List of DNS options. + +e.g. + +```json +"dns" { + "servers": ["10.0.0.10"], + "searches": ["default.svc.cluster.local","svc.cluster.local","cluster.local"], + "options": [] +} +``` + #### For 3. Open vSwitch (OVS) & Open Virtual Network (OVN) with Overlay {{< note >}} @@ -360,7 +387,7 @@ Some of these limitations will be addressed by the community in future releases - Mount propagation is not supported on Windows - The StatefulSet functionality for stateful applications is not supported - Horizontal Pod Autoscaling for Windows Server Container pods has not been verified to work end-to-end -- Hyper-V isolated containers are not supported. +- Hyper-V isolated containers are not supported. - Windows container OS must match the Host OS. If it does not, the pod will get stuck in a crash loop. - Under the networking models of L3 or Host GW, Kubernetes Services are inaccessible to Windows nodes due to a Windows issue. This is not an issue if using OVN/OVS for networking. - Windows kubelet.exe may fail to start when running on Windows Server under VMware Fusion [issue 57110](https://github.com/kubernetes/kubernetes/pull/57124) @@ -372,4 +399,3 @@ Some of these limitations will be addressed by the community in future releases - Support for Windows is in Beta as of v1.9 and your feedback is welcome. For information on getting involved, please head to [SIG-Windows](https://github.com/kubernetes/community/blob/master/sig-windows/README.md) - Troubleshooting and Common Problems: [Link](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/common-problems) - diff --git a/content/en/docs/reference/_index.md b/content/en/docs/reference/_index.md index b5e825bc7f..1ebaf37cca 100644 --- a/content/en/docs/reference/_index.md +++ b/content/en/docs/reference/_index.md @@ -20,6 +20,7 @@ This section of the Kubernetes documentation contains references. * [Kubernetes API Overview](/docs/reference/using-api/api-overview/) - Overview of the API for Kubernetes. * Kubernetes API Versions + * [1.13](/docs/reference/generated/kubernetes-api/v1.13/) * [1.12](/docs/reference/generated/kubernetes-api/v1.12/) * [1.11](/docs/reference/generated/kubernetes-api/v1.11/) * [1.10](https://v1-10.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/) diff --git a/content/en/docs/reference/access-authn-authz/admission-controllers.md b/content/en/docs/reference/access-authn-authz/admission-controllers.md index 95a29497eb..5dc9f2c311 100644 --- a/content/en/docs/reference/access-authn-authz/admission-controllers.md +++ b/content/en/docs/reference/access-authn-authz/admission-controllers.md @@ -415,6 +415,25 @@ This admission controller limits the `Node` and `Pod` objects a kubelet can modi kubelets must use credentials in the `system:nodes` group, with a username in the form `system:node:`. Such kubelets will only be allowed to modify their own `Node` API object, and only modify `Pod` API objects that are bound to their node. In Kubernetes 1.11+, kubelets are not allowed to update or remove taints from their `Node` API object. + +In Kubernetes 1.13+, the `NodeRestriction` admission plugin prevents kubelets from deleting their `Node` API object, +and enforces kubelet modification of labels under the `kubernetes.io/` or `k8s.io/` prefixes as follows: + +* **Prevents** kubelets from adding/removing/updating labels with a `node-restriction.kubernetes.io/` prefix. +This label prefix is reserved for administrators to label their `Node` objects for workload isolation purposes, +and kubelets will not be allowed to modify labels with that prefix. +* **Allows** kubelets to add/remove/update these labels and label prefixes: + * `kubernetes.io/hostname` + * `beta.kubernetes.io/arch` + * `beta.kubernetes.io/instance-type` + * `beta.kubernetes.io/os` + * `failure-domain.beta.kubernetes.io/region` + * `failure-domain.beta.kubernetes.io/zone` + * `kubelet.kubernetes.io/`-prefixed labels + * `node.kubernetes.io/`-prefixed labels + +Use of any other labels under the `kubernetes.io` or `k8s.io` prefixes by kubelets is reserved, and may be disallowed or allowed by the `NodeRestriction` admission plugin in the future. + Future versions may add additional restrictions to ensure kubelets have the minimal set of permissions required to operate correctly. ### OwnerReferencesPermissionEnforcement {#ownerreferencespermissionenforcement} diff --git a/content/en/docs/reference/access-authn-authz/rbac.md b/content/en/docs/reference/access-authn-authz/rbac.md index 719164c5ec..b90509b924 100644 --- a/content/en/docs/reference/access-authn-authz/rbac.md +++ b/content/en/docs/reference/access-authn-authz/rbac.md @@ -644,11 +644,13 @@ These roles include: * system:controller:attachdetach-controller * system:controller:certificate-controller +* system:controller:clusterrole-aggregation-controller * system:controller:cronjob-controller * system:controller:daemon-set-controller * system:controller:deployment-controller * system:controller:disruption-controller * system:controller:endpoint-controller +* system:controller:expand-controller * system:controller:generic-garbage-collector * system:controller:horizontal-pod-autoscaler * system:controller:job-controller @@ -661,6 +663,7 @@ These roles include: * system:controller:replicaset-controller * system:controller:replication-controller * system:controller:resourcequota-controller +* system:controller:root-ca-cert-publisher * system:controller:route-controller * system:controller:service-account-controller * system:controller:service-controller diff --git a/content/en/docs/reference/command-line-tools-reference/cloud-controller-manager.md b/content/en/docs/reference/command-line-tools-reference/cloud-controller-manager.md index 4ddb32c0db..cf9efcbaa5 100644 --- a/content/en/docs/reference/command-line-tools-reference/cloud-controller-manager.md +++ b/content/en/docs/reference/command-line-tools-reference/cloud-controller-manager.md @@ -32,6 +32,13 @@ cloud-controller-manager [flags] Should CIDRs for Pods be allocated and set on the cloud provider. + + --alsologtostderr + + + log to standard error as well as files + + --authentication-kubeconfig string @@ -96,7 +103,7 @@ cloud-controller-manager [flags] - --cert-dir string     Default: "/var/run/kubernetes" + --cert-dir string The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. @@ -130,6 +137,13 @@ cloud-controller-manager [flags] The provider for cloud services. Empty string for no provider. + + --cloud-provider-gce-lb-src-cidrs cidrs     Default: 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16 + + + CIDRs opened in GCE firewall for LB traffic proxy & health checks + + --cluster-cidr string @@ -183,7 +197,7 @@ cloud-controller-manager [flags] --feature-gates mapStringBool - A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=false)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CSIPersistentVolume=true|false (BETA - default=true)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GCERegionalPersistentDisk=true|false (BETA - default=true)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPluginsWatcher=true|false (BETA - default=true)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (ALPHA - default=false)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
VolumeScheduling=true|false (BETA - default=true)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) + A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (BETA - default=true)
BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
CustomResourceWebhookConversion=true|false (ALPHA - default=false)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (BETA - default=true)
DynamicAuditing=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPodResources=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (ALPHA - default=false)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) @@ -263,6 +277,27 @@ cloud-controller-manager [flags] The duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if leader election is enabled. + + --log-backtrace-at traceLocation     Default: :0 + + + when logging hits line file:N, emit a stack trace + + + + --log-dir string + + + If non-empty, write log files in this directory + + + + --log-file string + + + If non-empty, use this log file + + --log-flush-frequency duration     Default: 5s @@ -270,6 +305,13 @@ cloud-controller-manager [flags] Maximum number of seconds between log flushes + + --logtostderr     Default: true + + + log to standard error instead of files + + --master string @@ -354,6 +396,20 @@ cloud-controller-manager [flags] The port on which to serve HTTPS with authentication and authorization.If 0, don't serve HTTPS at all. + + --skip-headers + + + If true, avoid header prefixes in the log messages + + + + --stderrthreshold severity     Default: 2 + + + logs at or above this threshold go to stderr + + --tls-cert-file string @@ -396,6 +452,13 @@ cloud-controller-manager [flags] If true, use individual service account credentials for each controller. + + -v, --v Level + + + log level for V logs + + --version version[=true] @@ -403,6 +466,13 @@ cloud-controller-manager [flags] Print version information and quit + + --vmodule moduleSpec + + + comma-separated list of pattern=N settings for file-filtered logging + + diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md index 9726848ee2..7d665737c3 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md @@ -41,6 +41,8 @@ different Kubernetes components. | `AttachVolumeLimit` | `true` | Alpha | 1.11 | 1.11 | | `AttachVolumeLimit` | `true` | Beta | 1.12 | | | `BlockVolume` | `false` | Alpha | 1.9 | | +| `BlockVolume` | `true` | Beta | 1.13 | - | +| `BoundServiceAccountTokenVolume` | `false` | Alpha | 1.13 | | | `CPUCFSQuotaPeriod` | `false` | Alpha | 1.12 | | | `CPUManager` | `false` | Alpha | 1.8 | 1.9 | | `CPUManager` | `true` | Beta | 1.10 | | @@ -50,7 +52,8 @@ different Kubernetes components. | `CSIDriverRegistry` | `false` | Alpha | 1.12 | | | `CSINodeInfo` | `false` | Alpha | 1.12 | | | `CSIPersistentVolume` | `false` | Alpha | 1.9 | 1.9 | -| `CSIPersistentVolume` | `true` | Beta | 1.10 | | +| `CSIPersistentVolume` | `true` | Beta | 1.10 | 1.12 | +| `CSIPersistentVolume` | `true` | GA | 1.13 | - | | `CustomPodDNS` | `false` | Alpha | 1.9 | 1.9 | | `CustomPodDNS` | `true` | Beta| 1.10 | | | `CustomResourceSubresources` | `false` | Alpha | 1.10 | | @@ -59,6 +62,7 @@ different Kubernetes components. | `DebugContainers` | `false` | Alpha | 1.10 | | | `DevicePlugins` | `false` | Alpha | 1.8 | 1.9 | | `DevicePlugins` | `true` | Beta | 1.10 | | +| `DryRun` | `true` | Beta | 1.13 | | | `DynamicAuditing` | `false` | Alpha | 1.13 | | | `DynamicKubeletConfig` | `false` | Alpha | 1.4 | 1.10 | | `DynamicKubeletConfig` | `true` | Beta | 1.11 | | @@ -71,14 +75,17 @@ different Kubernetes components. | `ExpandPersistentVolumes` | `true` | Beta | 1.11 | | | `ExperimentalCriticalPodAnnotation` | `false` | Alpha | 1.5 | | | `ExperimentalHostUserNamespaceDefaulting` | `false` | Beta | 1.5 | | -| `GCERegionalPersistentDisk` | `true` | Beta | 1.10 | | +| `GCERegionalPersistentDisk` | `true` | Beta | 1.10 | 1.12 | +| `GCERegionalPersistentDisk` | `true` | GA | 1.13 | - | | `HugePages` | `false` | Alpha | 1.8 | 1.9 | | `HugePages` | `true` | Beta| 1.10 | | | `HyperVContainer` | `false` | Alpha | 1.10 | | | `Initializers` | `false` | Alpha | 1.7 | | | `KubeletConfigFile` | `false` | Alpha | 1.8 | 1.9 | | `KubeletPluginsWatcher` | `false` | Alpha | 1.11 | 1.11 | -| `KubeletPluginsWatcher` | `true` | Beta | 1.12 | | +| `KubeletPluginsWatcher` | `true` | Beta | 1.12 | 1.12 | +| `KubeletPluginsWatcher` | `true` | GA | 1.13 | - | +| `KubeletPodResources` | `false` | Alpha | 1.13 | | | `LocalStorageCapacityIsolation` | `false` | Alpha | 1.7 | 1.9 | | `LocalStorageCapacityIsolation` | `true` | Beta| 1.10 | | | `MountContainers` | `false` | Alpha | 1.9 | | @@ -114,18 +121,21 @@ different Kubernetes components. | `SupportIPVSProxyMode` | `true` | GA | 1.11 | | | `SupportPodPidsLimit` | `false` | Alpha | 1.10 | | | `Sysctls` | `true` | Beta | 1.11 | | -| `TaintBasedEvictions` | `false` | Alpha | 1.6 | | -| `TaintNodesByCondition` | `false` | Alpha | 1.8 | | +| `TaintBasedEvictions` | `false` | Alpha | 1.6 | 1.12 | +| `TaintBasedEvictions` | `true` | Beta | 1.13 | | +| `TaintNodesByCondition` | `false` | Alpha | 1.8 | 1.11 | | `TaintNodesByCondition` | `true` | Beta | 1.12 | | | `TokenRequest` | `false` | Alpha | 1.10 | 1.11 | -| `TokenRequest` | `True` | Beta | 1.12 | | +| `TokenRequest` | `true` | Beta | 1.12 | | | `TokenRequestProjection` | `false` | Alpha | 1.11 | 1.11 | -| `TokenRequestProjection` | `True` | Beta | 1.12 | | +| `TokenRequestProjection` | `true` | Beta | 1.12 | | | `TTLAfterFinished` | `false` | Alpha | 1.12 | | | `VolumeScheduling` | `false` | Alpha | 1.9 | 1.9 | -| `VolumeScheduling` | `true` | Beta | 1.10 | | +| `VolumeScheduling` | `true` | Beta | 1.10 | 1.12 | +| `VolumeScheduling` | `true` | GA | 1.13 | | | `VolumeSubpathEnvExpansion` | `false` | Alpha | 1.11 | | -| `VolumeSnapshotDataSource` | `false` | Alpha | 1.12 | | +| `VolumeSnapshotDataSource` | `false` | Alpha | 1.12 | - | +| `ScheduleDaemonSetPods` | `false` | Alpha | 1.11 | 1.11 | | `ScheduleDaemonSetPods` | `true` | Beta | 1.12 | | ## Using a Feature @@ -184,6 +194,10 @@ Each feature gate is designed for enabling/disabling a specific feature: - `BlockVolume`: Enable the definition and consumption of raw block devices in Pods. See [Raw Block Volume Support](/docs/concepts/storage/persistent-volumes/#raw-block-volume-support) for more details. +- `BoundServiceAccountTokenVolume`: Migrate ServiceAccount volumes to use a projected volume consisting of a + ServiceAccountTokenVolumeProjection. + Check [Service Account Token Volumes](https://git.k8s.io/community/contributors/design-proposals/storage/svcacct-token-volume-source.md) + for more details. - `CPUCFSQuotaPeriod`: Enable nodes to change CPUCFSQuotaPeriod. - `CPUManager`: Enable container level CPU affinity support, see [CPU Management Policies](/docs/tasks/administer-cluster/cpu-management-policies/). - `CRIContainerLogRotation`: Enable container log rotation for cri container runtime. @@ -205,6 +219,7 @@ Each feature gate is designed for enabling/disabling a specific feature: troubleshoot a running Pod. - `DevicePlugins`: Enable the [device-plugins](/docs/concepts/cluster-administration/device-plugins/) based resource provisioning on nodes. +- `DryRun`: Enable server-side [dry run](/docs/reference/using-api/api-concepts/#dry-run) requests. - `DynamicAuditing`: Enable [dynamic auditing](/docs/tasks/debug-application-cluster/audit/#dynamic-backend) - `DynamicKubeletConfig`: Enable the dynamic configuration of kubelet. See [Reconfigure kubelet](/docs/tasks/administer-cluster/reconfigure-kubelet/). - `DynamicProvisioningScheduling`: Extend the default scheduler to be aware of volume topology and handle PV provisioning. @@ -229,6 +244,8 @@ Each feature gate is designed for enabling/disabling a specific feature: See [setting kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file/) for more details. - `KubeletPluginsWatcher`: Enable probe-based plugin watcher utility to enable kubelet to discover plugins such as [CSI volume drivers](/docs/concepts/storage/volumes/#csi). +- `KubeletPodResources`: Enable the kubelet's pod resources grpc endpoint. + See [Support Device Monitoring](https://git.k8s.io/community/keps/sig-node/compute-device-assignment.md) for more details. - `LocalStorageCapacityIsolation`: Enable the consumption of [local ephemeral storage](/docs/concepts/configuration/manage-compute-resources-container/) and also the `sizeLimit` property of an [emptyDir volume](/docs/concepts/storage/volumes/#emptydir). - `MountContainers`: Enable using utility containers on host as the volume mounter. - `MountPropagation`: Enable sharing volume mounted by one container to other containers or pods. diff --git a/content/en/docs/reference/command-line-tools-reference/federation-apiserver.md b/content/en/docs/reference/command-line-tools-reference/federation-apiserver.md index 9a3e7137b2..c219528332 100644 --- a/content/en/docs/reference/command-line-tools-reference/federation-apiserver.md +++ b/content/en/docs/reference/command-line-tools-reference/federation-apiserver.md @@ -68,7 +68,7 @@ federation-apiserver [flags] --etcd-servers strings List of etcd servers to connect with (scheme://ip:port), comma separated. --etcd-servers-overrides strings Per-resource etcd servers overrides, comma separated. The individual override format: group/resource#servers, where servers are http://ip:port, semicolon separated. --event-ttl duration Amount of time to retain events. (default 1h0m0s) - --experimental-encryption-provider-config string The file containing configuration for encryption providers to be used for storing secrets in etcd + --encryption-provider-config string The file containing configuration for encryption providers to be used for storing secrets in etcd --experimental-keystone-ca-file string If set, the Keystone server's certificate will be verified by one of the authorities in the experimental-keystone-ca-file, otherwise the host's root CA set will be used. --experimental-keystone-url string If passed, activates the keystone authentication plugin. --external-hostname string The hostname to use when generating externalized URLs for this master (e.g. Swagger API Docs). @@ -150,4 +150,4 @@ VolumeScheduling=true|false (ALPHA - default=false) --watch-cache-sizes strings List of watch cache sizes for every resource (pods, nodes, etc.), comma separated. The individual override format: resource[.group]#size, where resource is lowercase plural (no version), group is optional, and size is a number. It takes effect when watch-cache is enabled. Some resources (replicationcontrollers, endpoints, nodes, pods, services, apiservices.apiregistration.k8s.io) have system defaults set by heuristics, others default to default-watch-cache-size ``` -###### Auto generated by spf13/cobra on 24-Sep-2018 +###### Auto generated by spf13/cobra on 1-Dec-2018 diff --git a/content/en/docs/reference/command-line-tools-reference/federation-controller-manager.md b/content/en/docs/reference/command-line-tools-reference/federation-controller-manager.md index e7d0460e55..1e48b2cbec 100644 --- a/content/en/docs/reference/command-line-tools-reference/federation-controller-manager.md +++ b/content/en/docs/reference/command-line-tools-reference/federation-controller-manager.md @@ -32,7 +32,7 @@ federation-controller-manager [flags] --concurrent-service-syncs int The number of service syncing operations that will be done concurrently. Larger number = faster endpoint updating, but more CPU (and network) load (default 10) --contention-profiling Enable lock contention profiling, if profiling is enabled --controllers mapStringString A set of key=value pairs that describe controller configuration to enable/disable specific controllers. Key should be the resource name (like services) and value should be true or false. For example: services=false,ingresses=false - --dns-provider string DNS provider. Valid values are: ["aws-route53" "azure-azuredns" "coredns" "google-clouddns"] + --dns-provider string DNS provider. Valid values are: ["coredns" "google-clouddns" "aws-route53" "azure-azuredns"] --dns-provider-config string Path to config file for configuring DNS provider. --federated-api-burst int Burst to use while talking with federation apiserver (default 30) --federated-api-qps float32 QPS to use while talking with federation apiserver (default 20) @@ -56,4 +56,4 @@ federation-controller-manager [flags] --zone-name string Zone name, like example.com. ``` -###### Auto generated by spf13/cobra on 24-Sep-2018 +###### Auto generated by spf13/cobra on 1-Dec-2018 diff --git a/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md b/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md index 0bc8a51723..73359207da 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md @@ -48,6 +48,13 @@ kube-apiserver [flags] If true, allow privileged containers. [default=false] + + --alsologtostderr + + + log to standard error as well as files + + --anonymous-auth     Default: true @@ -55,6 +62,13 @@ kube-apiserver [flags] Enables anonymous requests to the secure port of the API server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of system:anonymous, and a group name of system:unauthenticated. + + --api-audiences stringSlice + + + Identifiers of the API. The service account token authenticator will validate that tokens used against the API are bound to at least one of these audiences. If the --service-account-issuer flag is configured and this flag is not, this field defaults to a single element list containing the issuer URL . + + --apiserver-count int     Default: 1 @@ -62,6 +76,13 @@ kube-apiserver [flags] The number of apiservers running in the cluster, must be a positive number. (In use when --endpoint-reconciler-type=master-count is enabled.) + + --audit-dynamic-configuration + + + Enables dynamic audit configuration. This feature also requires the DynamicAuditing feature flag + + --audit-log-batch-buffer-size int     Default: 10000 @@ -136,7 +157,7 @@ kube-apiserver [flags] --audit-log-mode string     Default: "blocking" - Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking. + Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking,blocking-strict. @@ -168,7 +189,7 @@ kube-apiserver [flags] - --audit-log-version string     Default: "audit.k8s.io/v1beta1" + --audit-log-version string     Default: "audit.k8s.io/v1" API group and version used for serializing audit events written to log. @@ -241,7 +262,7 @@ kube-apiserver [flags] --audit-webhook-mode string     Default: "batch" - Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking. + Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking,blocking-strict. @@ -266,7 +287,7 @@ kube-apiserver [flags] - --audit-webhook-version string     Default: "audit.k8s.io/v1beta1" + --audit-webhook-version string     Default: "audit.k8s.io/v1" API group and version used for serializing audit events written to webhook. @@ -370,6 +391,13 @@ kube-apiserver [flags] The provider for cloud services. Empty string for no provider. + + --cloud-provider-gce-lb-src-cidrs cidrs     Default: 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16 + + + CIDRs opened in GCE firewall for LB traffic proxy & health checks + + --contention-profiling @@ -384,6 +412,20 @@ kube-apiserver [flags] List of allowed origins for CORS, comma separated. An allowed origin can be a regular expression to support subdomain matching. If this list is empty CORS will not be enabled. + + --default-not-ready-toleration-seconds int     Default: 300 + + + Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration. + + + + --default-unreachable-toleration-seconds int     Default: 300 + + + Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration. + + --default-watch-cache-size int     Default: 100 @@ -398,13 +440,6 @@ kube-apiserver [flags] Number of workers spawned for DeleteCollection call. These are used to speed up namespace cleanup. - - --deserialization-cache-size int - - - Number of deserialized json objects to cache in memory. - - --disable-admission-plugins stringSlice @@ -454,6 +489,13 @@ kube-apiserver [flags] Enables swagger ui on the apiserver at /swagger-ui + + --encryption-provider-config string + + + The file containing configuration for encryption providers to be used for storing secrets in etcd + + --endpoint-reconciler-type string     Default: "lease" @@ -525,13 +567,16 @@ kube-apiserver [flags] - --experimental-encryption-provider-config string +<<<<<<< HEAD + --encryption-provider-config string The file containing configuration for encryption providers to be used for storing secrets in etcd +======= +>>>>>>> Generate copmonents and tools reference --external-hostname string @@ -542,7 +587,7 @@ kube-apiserver [flags] --feature-gates mapStringBool - A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=false)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CSIPersistentVolume=true|false (BETA - default=true)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GCERegionalPersistentDisk=true|false (BETA - default=true)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPluginsWatcher=true|false (BETA - default=true)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (ALPHA - default=false)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
VolumeScheduling=true|false (BETA - default=true)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) + A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (BETA - default=true)
BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
CustomResourceWebhookConversion=true|false (ALPHA - default=false)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (BETA - default=true)
DynamicAuditing=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPodResources=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (ALPHA - default=false)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) @@ -615,6 +660,27 @@ kube-apiserver [flags] If non-zero, the Kubernetes master service (which apiserver creates/maintains) will be of type NodePort, using this as the value of the port. If zero, the Kubernetes master service will be of type ClusterIP. + + --log-backtrace-at traceLocation     Default: :0 + + + when logging hits line file:N, emit a stack trace + + + + --log-dir string + + + If non-empty, write log files in this directory + + + + --log-file string + + + If non-empty, use this log file + + --log-flush-frequency duration     Default: 5s @@ -622,6 +688,13 @@ kube-apiserver [flags] Maximum number of seconds between log flushes + + --logtostderr     Default: true + + + log to standard error instead of files + + --master-service-namespace string     Default: "default" @@ -797,13 +870,6 @@ kube-apiserver [flags] The port on which to serve HTTPS with authentication and authorization.It cannot be switched off with 0. - - --service-account-api-audiences stringSlice - - - Identifiers of the API. The service account token authenticator will validate that tokens used against the API are bound to at least one of these audiences. - - --service-account-issuer string @@ -853,11 +919,29 @@ kube-apiserver [flags] A port range to reserve for services with NodePort visibility. Example: '30000-32767'. Inclusive at both ends of the range. + + --skip-headers + + + If true, avoid header prefixes in the log messages + + + + --stderrthreshold severity     Default: 2 + + + logs at or above this threshold go to stderr + + --storage-backend string - The storage backend for persistence. Options: 'etcd3' (default), 'etcd2'. +<<<<<<< HEAD + The storage backend for persistence. Options: 'etcd3' (default) +======= + The storage backend for persistence. Options: 'etcd3' (default). +>>>>>>> Generate copmonents and tools reference @@ -916,6 +1000,13 @@ kube-apiserver [flags] If set, the file that will be used to secure the secure port of the API server via token authentication. + + -v, --v Level + + + log level for V logs + + --version version[=true] @@ -923,6 +1014,13 @@ kube-apiserver [flags] Print version information and quit + + --vmodule moduleSpec + + + comma-separated list of pattern=N settings for file-filtered logging + + --watch-cache     Default: true diff --git a/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md b/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md index 55d4404c6c..2f80707248 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md @@ -38,6 +38,13 @@ kube-controller-manager [flags] Should CIDRs for Pods be allocated and set on the cloud provider. + + --alsologtostderr + + + log to standard error as well as files + + --attach-detach-reconcile-sync-period duration     Default: 1m0s @@ -109,7 +116,7 @@ kube-controller-manager [flags] - --cert-dir string     Default: "/var/run/kubernetes" + --cert-dir string The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. @@ -143,6 +150,13 @@ kube-controller-manager [flags] The provider for cloud services. Empty string for no provider. + + --cloud-provider-gce-lb-src-cidrs cidrs     Default: 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16 + + + CIDRs opened in GCE firewall for LB traffic proxy & health checks + + --cluster-cidr string @@ -266,7 +280,7 @@ kube-controller-manager [flags] --controllers stringSlice     Default: [*] - A list of controllers to enable. '*' enables all on-by-default controllers, 'foo' enables the controller named 'foo', '-foo' disables the controller named 'foo'.
All controllers: attachdetach, bootstrapsigner, clusterrole-aggregation, cronjob, csrapproving, csrcleaner, csrsigning, daemonset, deployment, disruption, endpoint, garbagecollector, horizontalpodautoscaling, job, namespace, nodeipam, nodelifecycle, persistentvolume-binder, persistentvolume-expander, podgc, pv-protection, pvc-protection, replicaset, replicationcontroller, resourcequota, route, service, serviceaccount, serviceaccount-token, statefulset, tokencleaner, ttl, ttl-after-finished
Disabled-by-default controllers: bootstrapsigner, tokencleaner + A list of controllers to enable. '*' enables all on-by-default controllers, 'foo' enables the controller named 'foo', '-foo' disables the controller named 'foo'.
All controllers: attachdetach, bootstrapsigner, clusterrole-aggregation, cronjob, csrapproving, csrcleaner, csrsigning, daemonset, deployment, disruption, endpoint, garbagecollector, horizontalpodautoscaling, job, namespace, nodeipam, nodelifecycle, persistentvolume-binder, persistentvolume-expander, podgc, pv-protection, pvc-protection, replicaset, replicationcontroller, resourcequota, root-ca-cert-publisher, route, service, serviceaccount, serviceaccount-token, statefulset, tokencleaner, ttl, ttl-after-finished
Disabled-by-default controllers: bootstrapsigner, tokencleaner @@ -329,7 +343,7 @@ kube-controller-manager [flags] --feature-gates mapStringBool - A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=false)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CSIPersistentVolume=true|false (BETA - default=true)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GCERegionalPersistentDisk=true|false (BETA - default=true)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPluginsWatcher=true|false (BETA - default=true)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (ALPHA - default=false)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
VolumeScheduling=true|false (BETA - default=true)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) + A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (BETA - default=true)
BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
CustomResourceWebhookConversion=true|false (ALPHA - default=false)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (BETA - default=true)
DynamicAuditing=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPodResources=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (ALPHA - default=false)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) @@ -388,13 +402,6 @@ kube-controller-manager [flags] The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default. - - --insecure-experimental-approve-all-kubelet-csrs-for-group string - - - This flag does nothing. - - --kube-api-burst int32     Default: 30 @@ -465,6 +472,27 @@ kube-controller-manager [flags] The duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if leader election is enabled. + + --log-backtrace-at traceLocation     Default: :0 + + + when logging hits line file:N, emit a stack trace + + + + --log-dir string + + + If non-empty, write log files in this directory + + + + --log-file string + + + If non-empty, use this log file + + --log-flush-frequency duration     Default: 5s @@ -472,6 +500,13 @@ kube-controller-manager [flags] Maximum number of seconds between log flushes + + --logtostderr     Default: true + + + log to standard error instead of files + + --master string @@ -675,6 +710,20 @@ kube-controller-manager [flags] CIDR Range for Services in cluster. Requires --allocate-node-cidrs to be true + + --skip-headers + + + If true, avoid header prefixes in the log messages + + + + --stderrthreshold severity     Default: 2 + + + logs at or above this threshold go to stderr + + --terminated-pod-gc-threshold int32     Default: 12500 @@ -731,6 +780,13 @@ kube-controller-manager [flags] If true, use individual service account credentials for each controller. + + -v, --v Level + + + log level for V logs + + --version version[=true] @@ -738,6 +794,13 @@ kube-controller-manager [flags] Print version information and quit + + --vmodule moduleSpec + + + comma-separated list of pattern=N settings for file-filtered logging + + diff --git a/content/en/docs/reference/command-line-tools-reference/kube-proxy.md b/content/en/docs/reference/command-line-tools-reference/kube-proxy.md index 802edd5169..209e694c29 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-proxy.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-proxy.md @@ -111,7 +111,7 @@ kube-proxy [flags] --feature-gates mapStringBool - A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=false)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CSIPersistentVolume=true|false (BETA - default=true)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GCERegionalPersistentDisk=true|false (BETA - default=true)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPluginsWatcher=true|false (BETA - default=true)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (ALPHA - default=false)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
VolumeScheduling=true|false (BETA - default=true)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) + A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (BETA - default=true)
BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
CustomResourceWebhookConversion=true|false (ALPHA - default=false)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (BETA - default=true)
DynamicAuditing=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPodResources=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (ALPHA - default=false)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) diff --git a/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md b/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md index 0b6b79f413..511ee18e51 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md @@ -50,6 +50,62 @@ kube-scheduler [flags] DEPRECATED: the scheduling algorithm provider to use, one of: ClusterAutoscalerProvider | DefaultProvider + + --alsologtostderr + + + log to standard error as well as files + + + + --authentication-kubeconfig string + + + kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenaccessreviews.authentication.k8s.io. This is optional. If empty, all token requests are considered to be anonymous and no client CA is looked up in the cluster. + + + + --authentication-skip-lookup + + + If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster. + + + + --authentication-token-webhook-cache-ttl duration     Default: 10s + + + The duration to cache responses from the webhook token authenticator. + + + + --authorization-always-allow-paths stringSlice     Default: [/healthz] + + + A list of HTTP paths to skip during authorization, i.e. these are authorized without contacting the 'core' kubernetes server. + + + + --authorization-kubeconfig string + + + kubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io. This is optional. If empty, all requests not skipped by authorization are forbidden. + + + + --authorization-webhook-cache-authorized-ttl duration     Default: 10s + + + The duration to cache 'authorized' responses from the webhook authorizer. + + + + --authorization-webhook-cache-unauthorized-ttl duration     Default: 10s + + + The duration to cache 'unauthorized' responses from the webhook authorizer. + + --azure-container-registry-config string @@ -57,6 +113,27 @@ kube-scheduler [flags] Path to the file containing Azure container registry configuration information. + + --bind-address ip     Default: 0.0.0.0 + + + The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank, all interfaces will be used (0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces). + + + + --cert-dir string + + + The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. + + + + --client-ca-file string + + + If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate. + + --config string @@ -75,7 +152,7 @@ kube-scheduler [flags] --feature-gates mapStringBool - A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=false)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CSIPersistentVolume=true|false (BETA - default=true)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GCERegionalPersistentDisk=true|false (BETA - default=true)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPluginsWatcher=true|false (BETA - default=true)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (ALPHA - default=false)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
VolumeScheduling=true|false (BETA - default=true)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) + A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (BETA - default=true)
BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
CustomResourceWebhookConversion=true|false (ALPHA - default=false)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (BETA - default=true)
DynamicAuditing=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPodResources=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (ALPHA - default=false)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) @@ -85,6 +162,13 @@ kube-scheduler [flags] help for kube-scheduler + + --http2-max-streams-per-connection int + + + The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default. + + --kube-api-burst int32     Default: 100 @@ -162,6 +246,27 @@ kube-scheduler [flags] DEPRECATED: define the namespace of the lock object. + + --log-backtrace-at traceLocation     Default: :0 + + + when logging hits line file:N, emit a stack trace + + + + --log-dir string + + + If non-empty, write log files in this directory + + + + --log-file string + + + If non-empty, use this log file + + --log-flush-frequency duration     Default: 5s @@ -169,6 +274,13 @@ kube-scheduler [flags] Maximum number of seconds between log flushes + + --logtostderr     Default: true + + + log to standard error instead of files + + --master string @@ -218,6 +330,41 @@ kube-scheduler [flags] DEPRECATED: enable profiling via web interface host:port/debug/pprof/ + + --requestheader-allowed-names stringSlice + + + List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed. + + + + --requestheader-client-ca-file string + + + Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests. + + + + --requestheader-extra-headers-prefix stringSlice     Default: [x-remote-extra-] + + + List of request header prefixes to inspect. X-Remote-Extra- is suggested. + + + + --requestheader-group-headers stringSlice     Default: [x-remote-group] + + + List of request headers to inspect for groups. X-Remote-Group is suggested. + + + + --requestheader-username-headers stringSlice     Default: [x-remote-user] + + + List of request headers to inspect for usernames. X-Remote-User is common. + + --scheduler-name string     Default: "default-scheduler" @@ -225,6 +372,62 @@ kube-scheduler [flags] DEPRECATED: name of the scheduler, used to select which pods will be processed by this scheduler, based on pod's "spec.schedulerName". + + --secure-port int     Default: 10259 + + + The port on which to serve HTTPS with authentication and authorization.If 0, don't serve HTTPS at all. + + + + --skip-headers + + + If true, avoid header prefixes in the log messages + + + + --stderrthreshold severity     Default: 2 + + + logs at or above this threshold go to stderr + + + + --tls-cert-file string + + + File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir. + + + + --tls-cipher-suites stringSlice + + + Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be use. Possible values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_RC4_128_SHA + + + + --tls-min-version string + + + Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12 + + + + --tls-private-key-file string + + + File containing the default x509 private key matching --tls-cert-file. + + + + --tls-sni-cert-key namedCertKey     Default: [] + + + A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com". + + --use-legacy-policy-config @@ -232,6 +435,13 @@ kube-scheduler [flags] DEPRECATED: when set to true, scheduler will ignore policy ConfigMap and uses policy config file + + -v, --v Level + + + log level for V logs + + --version version[=true] @@ -239,6 +449,13 @@ kube-scheduler [flags] Print version information and quit + + --vmodule moduleSpec + + + comma-separated list of pattern=N settings for file-filtered logging + + --write-config-to string diff --git a/content/en/docs/reference/command-line-tools-reference/kubelet.md b/content/en/docs/reference/command-line-tools-reference/kubelet.md index 29a3ff6eea..90085458bf 100644 --- a/content/en/docs/reference/command-line-tools-reference/kubelet.md +++ b/content/en/docs/reference/command-line-tools-reference/kubelet.md @@ -545,7 +545,7 @@ kubelet [flags] --feature-gates mapStringBool - A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
Accelerators=true|false
AdvancedAuditing=true|false (BETA - default=true)
AllAlpha=true|false (ALPHA - default=false)
AllowExtTrafficLocalEndpoints=true|false
AppArmor=true|false (BETA - default=true)
BlockVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CSIPersistentVolume=true|false (ALPHA - default=false)
CustomPodDNS=true|false (ALPHA - default=false)
CustomResourceValidation=true|false (BETA - default=true)
DebugContainers=true|false
DevicePlugins=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (ALPHA - default=false)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (ALPHA - default=false)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletConfigFile=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (ALPHA - default=false)
MountContainers=true|false (ALPHA - default=false)
MountPropagation=true|false (ALPHA - default=false)
PVCProtection=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (ALPHA - default=false)
PodPriority=true|false (ALPHA - default=false)
ReadOnlyAPIDataVolumes=true|false
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (ALPHA - default=false)
ServiceNodeExclusion=true|false (ALPHA - default=false)
ServiceProxyAllowExternalIPs=true|false
StreamingProxyRedirects=true|false (BETA - default=true)
SupportIPVSProxyMode=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (ALPHA - default=false)
TaintNodesByCondition=true|false (ALPHA - default=false)
VolumeScheduling=true|false (ALPHA - default=false)
VolumeSubpath=true|false
+ A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
Accelerators=true|false
AdvancedAuditing=true|false (BETA - default=true)
AllAlpha=true|false (ALPHA - default=false)
AllowExtTrafficLocalEndpoints=true|false
AppArmor=true|false (BETA - default=true)
BlockVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CSIPersistentVolume=true|false (ALPHA - default=false)
CustomPodDNS=true|false (ALPHA - default=false)
CustomResourceValidation=true|false (BETA - default=true)
DebugContainers=true|false
DevicePlugins=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (ALPHA - default=false)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (ALPHA - default=false)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletConfigFile=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (ALPHA - default=false)
MountContainers=true|false (ALPHA - default=false)
MountPropagation=true|false (ALPHA - default=false)
PVCProtection=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (ALPHA - default=false)
PodPriority=true|false (ALPHA - default=false)
ReadOnlyAPIDataVolumes=true|false
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (ALPHA - default=false)
ServiceNodeExclusion=true|false (ALPHA - default=false)
ServiceProxyAllowExternalIPs=true|false
StreamingProxyRedirects=true|false (BETA - default=true)
SupportIPVSProxyMode=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
VolumeScheduling=true|false (ALPHA - default=false)
VolumeSubpath=true|false
diff --git a/content/en/docs/reference/federation/extensions/v1beta1/definitions.html b/content/en/docs/reference/federation/extensions/v1beta1/definitions.html index 85ce8eefde..333a1816cf 100755 --- a/content/en/docs/reference/federation/extensions/v1beta1/definitions.html +++ b/content/en/docs/reference/federation/extensions/v1beta1/definitions.html @@ -1451,7 +1451,7 @@ span.icon > [class^="icon-"], span.icon > [class*=" icon-"] { cursor: default; }

v1beta1.HTTPIngressRuleValue

-

HTTPIngressRuleValue is a list of http selectors pointing to backends. In the example: http://<host>/<path>?<searchpart> → backend where parts of the url correspond to RFC 3986, this resource will be used to match against everything after the last / and before the first ? or #.

+

HTTPIngressRuleValue is a list of http selectors pointing to backends. In the example: http://<host>/<path>?<searchpart> → backend where where parts of the url correspond to RFC 3986, this resource will be used to match against everything after the last / and before the first ? or #.

@@ -7589,4 +7589,4 @@ Both these may change in the future. Incoming requests are matched against the h - + \ No newline at end of file diff --git a/content/en/docs/reference/federation/extensions/v1beta1/operations.html b/content/en/docs/reference/federation/extensions/v1beta1/operations.html index 1818797217..5d49d5d12a 100755 --- a/content/en/docs/reference/federation/extensions/v1beta1/operations.html +++ b/content/en/docs/reference/federation/extensions/v1beta1/operations.html @@ -9929,4 +9929,4 @@ span.icon > [class^="icon-"], span.icon > [class*=" icon-"] { cursor: default; } - + \ No newline at end of file diff --git a/content/en/docs/reference/federation/federation/v1beta1/definitions.html b/content/en/docs/reference/federation/federation/v1beta1/definitions.html new file mode 100755 index 0000000000..f71ec6ce0f --- /dev/null +++ b/content/en/docs/reference/federation/federation/v1beta1/definitions.html @@ -0,0 +1,1606 @@ +--- +title: federation/v1beta1 Model Definitions +notitle: true +--- + + + + + + + +Top Level API Objects + + + + +
+
+

Top Level API Objects

+
+
+ +
+
+
+
+

Definitions

+
+
+

v1.APIResourceList

+
+

APIResourceList is a list of APIResource, it is used to expose the name of the resources supported in a specific group and version, and if the resource is namespaced.

+
+
+++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

kind

Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds

false

string

apiVersion

APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources

false

string

groupVersion

groupVersion is the group and version this APIResourceList is for.

true

string

resources

resources contains the name of the resources and if they are namespaced.

true

v1.APIResource array

+ +
+
+

v1.Patch

+
+

Patch is provided to give a concrete name and type to the Kubernetes PATCH request body.

+
+
+
+

v1.DeleteOptions

+
+

DeleteOptions may be provided when deleting an API object.

+
+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

kind

Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds

false

string

apiVersion

APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources

false

string

gracePeriodSeconds

The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.

false

integer (int64)

preconditions

Must be fulfilled before a deletion is carried out. If not possible, a 409 Conflict status will be returned.

false

v1.Preconditions

orphanDependents

Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object’s finalizers list. Either this field or PropagationPolicy may be set, but not both.

false

boolean

false

propagationPolicy

Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: Orphan - orphan the dependents; Background - allow the garbage collector to delete the dependents in the background; Foreground - a cascading policy that deletes all dependents in the foreground.

false

v1.DeletionPropagation

+ +
+
+

v1beta1.ServerAddressByClientCIDR

+
+

ServerAddressByClientCIDR helps the client to determine the server address that they should use, depending on the clientCIDR that they match.

+
+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

clientCIDR

The CIDR with which clients can match their IP to figure out the server address that they should use.

true

string

serverAddress

Address of this server, suitable for a client that matches the above CIDR. This can be a hostname, hostname:port, IP or IP:port.

true

string

+ +
+
+

v1beta1.ClusterList

+
+

A list of all the kubernetes clusters registered to the federation

+
+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

kind

Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds

false

string

apiVersion

APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources

false

string

metadata

Standard list metadata. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds

false

v1.ListMeta

items

List of Cluster objects.

true

v1beta1.Cluster array

+ +
+
+

v1.ListMeta

+
+

ListMeta describes metadata that synthetic resources must have, including lists and various status objects. A resource may have only one of {ObjectMeta, ListMeta}.

+
+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

selfLink

selfLink is a URL representing this object. Populated by the system. Read-only.

false

string

resourceVersion

String that identifies the server’s internal version of this object that can be used by clients to determine when objects have changed. Value must be treated as opaque by clients and passed unmodified back to the server. Populated by the system. Read-only. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#concurrency-control-and-consistency

false

string

continue

continue may be set if the user set a limit on the number of items returned, and indicates that the server has more data available. The value is opaque and may be used to issue another request to the endpoint that served this list to retrieve the next set of available objects. Continuing a list may not be possible if the server configuration has changed or more than a few minutes have passed. The resourceVersion field returned when using this continue value will be identical to the value in the first response.

false

string

+ +
+
+

v1.StatusDetails

+
+

StatusDetails is a set of additional properties that MAY be set by the server to provide additional information about a response. The Reason field of a Status object defines what attributes will be set. Clients must ignore fields that do not match the defined type of each attribute, and should assume that any attribute may be empty, invalid, or under defined.

+
+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

name

The name attribute of the resource associated with the status StatusReason (when there is a single name which can be described).

false

string

group

The group attribute of the resource associated with the status StatusReason.

false

string

kind

The kind attribute of the resource associated with the status StatusReason. On some operations may differ from the requested resource Kind. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds

false

string

uid

UID of the resource. (when there is a single resource which can be described). More info: http://kubernetes.io/docs/user-guide/identifiers#uids

false

string

causes

The Causes array includes more details associated with the StatusReason failure. Not all StatusReasons may provide detailed causes.

false

v1.StatusCause array

retryAfterSeconds

If specified, the time in seconds before the operation should be retried. Some errors may indicate the client must take an alternate action - for those errors this field may indicate how long to wait before taking the alternate action.

false

integer (int32)

+ +
+
+

v1beta1.ClusterStatus

+
+

ClusterStatus is information about the current status of a cluster updated by cluster controller periodically.

+
+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

conditions

Conditions is an array of current cluster conditions.

false

v1beta1.ClusterCondition array

zones

Zones is the list of availability zones in which the nodes of the cluster exist, e.g. us-east1-a. These will always be in the same region.

false

string array

region

Region is the name of the region in which all of the nodes in the cluster exist. e.g. us-east1.

false

string

+ +
+
+

v1.Preconditions

+
+

Preconditions must be fulfilled before an operation (update, delete, etc.) is carried out.

+
+ +++++++ + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

uid

Specifies the target UID.

false

types.UID

+ +
+
+

v1.Initializers

+
+

Initializers tracks the progress of initialization.

+
+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

pending

Pending is a list of initializers that must execute in order before this object is visible. When the last pending initializer is removed, and no failing result is set, the initializers struct will be set to nil and the object is considered as initialized and visible to all clients.

true

v1.Initializer array

result

If result is set with the Failure field, the object will be persisted to storage and then deleted, ensuring that other clients can observe the deletion.

false

v1.Status

+ +
+
+

v1.Initializer

+
+

Initializer is information about an initializer that has not yet completed.

+
+ +++++++ + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

name

name of the process that is responsible for initializing this object.

true

string

+ +
+
+

v1.LocalObjectReference

+
+

LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.

+
+ +++++++ + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

name

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

false

string

+ +
+
+

v1.Status

+
+

Status is a return value for calls that don’t return other objects.

+
+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

kind

Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds

false

string

apiVersion

APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources

false

string

metadata

Standard list metadata. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds

false

v1.ListMeta

status

Status of the operation. One of: "Success" or "Failure". More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status

false

string

message

A human-readable description of the status of this operation.

false

string

reason

A machine-readable description of why this operation is in the "Failure" status. If this value is empty there is no information available. A Reason clarifies an HTTP status code but does not override it.

false

string

details

Extended data associated with the reason. Each reason may define its own extended details. This field is optional and the data returned is not guaranteed to conform to any schema except that defined by the reason type.

false

v1.StatusDetails

code

Suggested HTTP return code for this status, 0 if not set.

false

integer (int32)

+ +
+
+

v1beta1.ClusterSpec

+
+

ClusterSpec describes the attributes of a kubernetes cluster.

+
+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

serverAddressByClientCIDRs

A map of client CIDR to server address. This is to help clients reach servers in the most network-efficient way possible. Clients can use the appropriate server address as per the CIDR that they match. In case of multiple matches, clients should use the longest matching CIDR.

true

v1beta1.ServerAddressByClientCIDR array

secretRef

Name of the secret containing kubeconfig to access this cluster. The secret is read from the kubernetes cluster that is hosting federation control plane. Admin needs to ensure that the required secret exists. Secret should be in the same namespace where federation control plane is hosted and it should have kubeconfig in its data with key "kubeconfig". This will later be changed to a reference to secret in federation control plane when the federation control plane supports secrets. This can be left empty if the cluster allows insecure access.

false

v1.LocalObjectReference

+ +
+
+

v1.WatchEvent

+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

type

true

string

object

true

string

+ +
+
+

v1beta1.ClusterCondition

+
+

ClusterCondition describes current state of a cluster.

+
+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

type

Type of cluster condition, Complete or Failed.

true

string

status

Status of the condition, one of True, False, Unknown.

true

string

lastProbeTime

Last time the condition was checked.

false

string

lastTransitionTime

Last time the condition transit from one status to another.

false

string

reason

(brief) reason for the condition’s last transition.

false

string

message

Human readable message indicating details about last transition.

false

string

+ +
+
+

v1.ObjectMeta

+
+

ObjectMeta is metadata that all persisted resources must have, which includes all objects users must create.

+
+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

name

Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/identifiers#names

false

string

generateName

GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.
+
+If this field is specified and the generated name exists, the server will NOT return a 409 - instead, it will either return 201 Created or 500 with Reason ServerTimeout indicating a unique name could not be found in the time allotted, and the client should retry (optionally after the time indicated in the Retry-After header).
+
+Applied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#idempotency

false

string

namespace

Namespace defines the space within each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.
+
+Must be a DNS_LABEL. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/namespaces

false

string

selfLink

SelfLink is a URL representing this object. Populated by the system. Read-only.

false

string

uid

UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.
+
+Populated by the system. Read-only. More info: http://kubernetes.io/docs/user-guide/identifiers#uids

false

string

resourceVersion

An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.
+
+Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#concurrency-control-and-consistency

false

string

generation

A sequence number representing a specific generation of the desired state. Populated by the system. Read-only.

false

integer (int64)

creationTimestamp

CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.
+
+Populated by the system. Read-only. Null for lists. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata

false

string

deletionTimestamp

DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.
+
+Populated by the system when a graceful deletion is requested. Read-only. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata

false

string

deletionGracePeriodSeconds

Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only.

false

integer (int64)

labels

Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: http://kubernetes.io/docs/user-guide/labels

false

object

annotations

Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: http://kubernetes.io/docs/user-guide/annotations

false

object

ownerReferences

List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller.

false

v1.OwnerReference array

initializers

An initializer is a controller which enforces some system invariant at object creation time. This field is a list of initializers that have not yet acted on this object. If nil or empty, this object has been completely initialized. Otherwise, the object is considered uninitialized and is hidden (in list/watch and get calls) from clients that haven’t explicitly asked to observe uninitialized objects.
+
+When an object is created, the system will populate this list with the current set of initializers. Only privileged users may set or modify this list. Once it is empty, it may not be modified further by any user.

false

v1.Initializers

finalizers

Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed.

false

string array

clusterName

The name of the cluster which the object belongs to. This is used to distinguish resources with same name and namespace in different clusters. This field is not set anywhere right now and apiserver is going to ignore it if set in create or update request.

false

string

+ +
+
+

v1.OwnerReference

+
+

OwnerReference contains enough information to let you identify an owning object. Currently, an owning object must be in the same namespace, so there is no namespace field.

+
+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

apiVersion

API version of the referent.

true

string

kind

Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds

true

string

name

Name of the referent. More info: http://kubernetes.io/docs/user-guide/identifiers#names

true

string

uid

UID of the referent. More info: http://kubernetes.io/docs/user-guide/identifiers#uids

true

string

controller

If true, this reference points to the managing controller.

false

boolean

false

blockOwnerDeletion

If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned.

false

boolean

false

+ +
+
+

v1.APIResource

+
+

APIResource specifies the name of a resource and whether it is namespaced.

+
+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

name

name is the plural name of the resource.

true

string

singularName

singularName is the singular name of the resource. This allows clients to handle plural and singular opaquely. The singularName is more correct for reporting status on a single item and both singular and plural are allowed from the kubectl CLI interface.

true

string

namespaced

namespaced indicates if a resource is namespaced or not.

true

boolean

false

group

group is the preferred group of the resource. Empty implies the group of the containing resource list. For subresources, this may have a different value, for example: Scale".

false

string

version

version is the preferred version of the resource. Empty implies the version of the containing resource list For subresources, this may have a different value, for example: v1 (while inside a v1beta1 version of the core resource’s group)".

false

string

kind

kind is the kind for the resource (e.g. Foo is the kind for a resource foo)

true

string

verbs

verbs is a list of supported kube verbs (this includes get, list, watch, create, update, patch, delete, deletecollection, and proxy)

true

string array

shortNames

shortNames is a list of suggested short names of the resource.

false

string array

categories

categories is a list of the grouped resources this resource belongs to (e.g. all)

false

string array

+ +
+
+

types.UID

+ +
+
+

v1.StatusCause

+
+

StatusCause provides more information about an api.Status failure, including cases when multiple errors are encountered.

+
+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

reason

A machine-readable description of the cause of the error. If this value is empty there is no information available.

false

string

message

A human-readable description of the cause of the error. This field may be presented as-is to a reader.

false

string

field

The field of the resource that has caused this error, as named by its JSON serialization. May include dot and postfix notation for nested attributes. Arrays are zero-indexed. Fields may appear more than once in an array of causes due to fields having multiple errors. Optional.
+
+Examples:
+ "name" - the field "name" on the current resource
+ "items[0].name" - the field "name" on the first array entry in "items"

false

string

+ +
+
+

v1.DeletionPropagation

+ +
+
+

v1beta1.Cluster

+
+

Information about a registered cluster in a federated kubernetes setup. Clusters are not namespaced and have unique names in the federation.

+
+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequiredSchemaDefault

kind

Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds

false

string

apiVersion

APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources

false

string

metadata

Standard object’s metadata. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata

false

v1.ObjectMeta

spec

Spec defines the behavior of the Cluster.

false

v1beta1.ClusterSpec

status

Status describes the current status of a Cluster

false

v1beta1.ClusterStatus

+ +
+
+

any

+
+

Represents an untyped JSON map - see the description of the field for more info about the structure of this object.

+
+
+ + + + + + \ No newline at end of file diff --git a/content/en/docs/reference/federation/federation/v1beta1/operations.html b/content/en/docs/reference/federation/federation/v1beta1/operations.html new file mode 100755 index 0000000000..360cba2503 --- /dev/null +++ b/content/en/docs/reference/federation/federation/v1beta1/operations.html @@ -0,0 +1,1921 @@ +--- +title: federation/v1beta1 Operations +notitle: true +--- + + + + + + + +Operations + + + + +
+
+

Operations

+
+
+

get available resources

+
+
+
GET /apis/federation/v1beta1
+
+
+
+

Responses

+ +++++ + + + + + + + + + + + + + + +
HTTP CodeDescriptionSchema

default

success

v1.APIResourceList

+ +
+
+

Consumes

+
+
    +
  • +

    application/json

    +
    • +
  • +

    application/yaml

    +
    • +
  • +

    application/vnd.kubernetes.protobuf

    +
    • +
+
+
+
+

Produces

+
+
    +
  • +

    application/json

    +
    • +
  • +

    application/yaml

    +
    • +
  • +

    application/vnd.kubernetes.protobuf

    +
    • +
+
+
+
+

Tags

+
+
    +
  • +

    apisfederationv1beta1

    +
    • +
+
+
+
+
+

list or watch objects of kind Cluster

+
+
+
GET /apis/federation/v1beta1/clusters
+
+
+
+

Parameters

+ ++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeNameDescriptionRequiredSchemaDefault

QueryParameter

pretty

If true, then the output is pretty printed.

false

string

QueryParameter

labelSelector

A selector to restrict the list of returned objects by their labels. Defaults to everything.

false

string

QueryParameter

fieldSelector

A selector to restrict the list of returned objects by their fields. Defaults to everything.

false

string

QueryParameter

includeUninitialized

If true, partially initialized resources are included in the response.

false

boolean

QueryParameter

watch

Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.

false

boolean

QueryParameter

resourceVersion

When specified with a watch call, shows changes that occur after that particular version of a resource. Defaults to changes from the beginning of history. When specified for list: - if unset, then the result is returned from remote storage based on quorum-read flag; - if it’s 0, then we simply return what we currently have in cache, no guarantee; - if set to non zero, then the result is at least as fresh as given rv.

false

string

QueryParameter

timeoutSeconds

Timeout for the list/watch call.

false

integer (int32)

QueryParameter

limit

limit is a maximum number of responses to return for a list call. If more items exist, the server will set the continue field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true. +

The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.

false

integer (int32)

QueryParameter

continue

The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server the server will respond with a 410 ResourceExpired error indicating the client must restart their list without the continue field. This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.

false

string

+ +
+
+

Responses

+ +++++ + + + + + + + + + + + + + + +
HTTP CodeDescriptionSchema

200

success

v1beta1.ClusterList

+ +
+
+

Consumes

+
+
    +
  • +

    /

    +
    • +
+
+
+
+

Produces

+
+
    +
  • +

    application/json

    +
    • +
  • +

    application/yaml

    +
    • +
  • +

    application/vnd.kubernetes.protobuf

    +
    • +
  • +

    application/json;stream=watch

    +
    • +
  • +

    application/vnd.kubernetes.protobuf;stream=watch

    +
    • +
+
+
+
+

Tags

+
+
    +
  • +

    apisfederationv1beta1

    +
    • +
+
+
+
+
+

delete collection of Cluster

+
+
+
DELETE /apis/federation/v1beta1/clusters
+
+
+
+

Parameters

+ ++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeNameDescriptionRequiredSchemaDefault

QueryParameter

pretty

If true, then the output is pretty printed.

false

string

QueryParameter

labelSelector

A selector to restrict the list of returned objects by their labels. Defaults to everything.

false

string

QueryParameter

fieldSelector

A selector to restrict the list of returned objects by their fields. Defaults to everything.

false

string

QueryParameter

includeUninitialized

If true, partially initialized resources are included in the response.

false

boolean

QueryParameter

watch

Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.

false

boolean

QueryParameter

resourceVersion

When specified with a watch call, shows changes that occur after that particular version of a resource. Defaults to changes from the beginning of history. When specified for list: - if unset, then the result is returned from remote storage based on quorum-read flag; - if it’s 0, then we simply return what we currently have in cache, no guarantee; - if set to non zero, then the result is at least as fresh as given rv.

false

string

QueryParameter

timeoutSeconds

Timeout for the list/watch call.

false

integer (int32)

QueryParameter

limit

limit is a maximum number of responses to return for a list call. If more items exist, the server will set the continue field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true. +

The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.

false

integer (int32)

QueryParameter

continue

The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server the server will respond with a 410 ResourceExpired error indicating the client must restart their list without the continue field. This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.

false

string

+ +
+
+

Responses

+ +++++ + + + + + + + + + + + + + + +
HTTP CodeDescriptionSchema

200

success

v1.Status

+ +
+
+

Consumes

+
+
    +
  • +

    /

    +
    • +
+
+
+
+

Produces

+
+
    +
  • +

    application/json

    +
    • +
  • +

    application/yaml

    +
    • +
  • +

    application/vnd.kubernetes.protobuf

    +
    • +
+
+
+
+

Tags

+
+
    +
  • +

    apisfederationv1beta1

    +
    • +
+
+
+
+
+

create a Cluster

+
+
+
POST /apis/federation/v1beta1/clusters
+
+
+
+

Parameters

+ ++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeNameDescriptionRequiredSchemaDefault

QueryParameter

pretty

If true, then the output is pretty printed.

false

string

BodyParameter

body

true

v1beta1.Cluster

+ +
+
+

Responses

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + +
HTTP CodeDescriptionSchema

202

Accepted

v1beta1.Cluster

200

success

v1beta1.Cluster

201

Created

v1beta1.Cluster

+ +
+
+

Consumes

+
+
    +
  • +

    /

    +
    • +
+
+
+
+

Produces

+
+
    +
  • +

    application/json

    +
    • +
  • +

    application/yaml

    +
    • +
  • +

    application/vnd.kubernetes.protobuf

    +
    • +
+
+
+
+

Tags

+
+
    +
  • +

    apisfederationv1beta1

    +
    • +
+
+
+
+
+

read the specified Cluster

+
+
+
GET /apis/federation/v1beta1/clusters/{name}
+
+
+
+

Parameters

+ ++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeNameDescriptionRequiredSchemaDefault

QueryParameter

pretty

If true, then the output is pretty printed.

false

string

QueryParameter

export

Should this value be exported. Export strips fields that a user can not specify.

false

boolean

QueryParameter

exact

Should the export be exact. Exact export maintains cluster-specific fields like Namespace.

false

boolean

PathParameter

name

name of the Cluster

true

string

+ +
+
+

Responses

+ +++++ + + + + + + + + + + + + + + +
HTTP CodeDescriptionSchema

200

success

v1beta1.Cluster

+ +
+
+

Consumes

+
+
    +
  • +

    /

    +
    • +
+
+
+
+

Produces

+
+
    +
  • +

    application/json

    +
    • +
  • +

    application/yaml

    +
    • +
  • +

    application/vnd.kubernetes.protobuf

    +
    • +
+
+
+
+

Tags

+
+
    +
  • +

    apisfederationv1beta1

    +
    • +
+
+
+
+
+

replace the specified Cluster

+
+
+
PUT /apis/federation/v1beta1/clusters/{name}
+
+
+
+

Parameters

+ ++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeNameDescriptionRequiredSchemaDefault

QueryParameter

pretty

If true, then the output is pretty printed.

false

string

BodyParameter

body

true

v1beta1.Cluster

PathParameter

name

name of the Cluster

true

string

+ +
+
+

Responses

+ +++++ + + + + + + + + + + + + + + + + + + + +
HTTP CodeDescriptionSchema

200

success

v1beta1.Cluster

201

Created

v1beta1.Cluster

+ +
+
+

Consumes

+
+
    +
  • +

    /

    +
    • +
+
+
+
+

Produces

+
+
    +
  • +

    application/json

    +
    • +
  • +

    application/yaml

    +
    • +
  • +

    application/vnd.kubernetes.protobuf

    +
    • +
+
+
+
+

Tags

+
+
    +
  • +

    apisfederationv1beta1

    +
    • +
+
+
+
+
+

delete a Cluster

+
+
+
DELETE /apis/federation/v1beta1/clusters/{name}
+
+
+
+

Parameters

+ ++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeNameDescriptionRequiredSchemaDefault

QueryParameter

pretty

If true, then the output is pretty printed.

false

string

BodyParameter

body

true

v1.DeleteOptions

QueryParameter

gracePeriodSeconds

The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.

false

integer (int32)

QueryParameter

orphanDependents

Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object’s finalizers list. Either this field or PropagationPolicy may be set, but not both.

false

boolean

QueryParameter

propagationPolicy

Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: Orphan - orphan the dependents; Background - allow the garbage collector to delete the dependents in the background; Foreground - a cascading policy that deletes all dependents in the foreground.

false

string

PathParameter

name

name of the Cluster

true

string

+ +
+
+

Responses

+ +++++ + + + + + + + + + + + + + + +
HTTP CodeDescriptionSchema

200

success

v1.Status

+ +
+
+

Consumes

+
+
    +
  • +

    /

    +
    • +
+
+
+
+

Produces

+
+
    +
  • +

    application/json

    +
    • +
  • +

    application/yaml

    +
    • +
  • +

    application/vnd.kubernetes.protobuf

    +
    • +
+
+
+
+

Tags

+
+
    +
  • +

    apisfederationv1beta1

    +
    • +
+
+
+
+
+

partially update the specified Cluster

+
+
+
PATCH /apis/federation/v1beta1/clusters/{name}
+
+
+
+

Parameters

+ ++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeNameDescriptionRequiredSchemaDefault

QueryParameter

pretty

If true, then the output is pretty printed.

false

string

BodyParameter

body

true

v1.Patch

PathParameter

name

name of the Cluster

true

string

+ +
+
+

Responses

+ +++++ + + + + + + + + + + + + + + +
HTTP CodeDescriptionSchema

200

success

v1beta1.Cluster

+ +
+
+

Consumes

+
+
    +
  • +

    application/json-patch+json

    +
    • +
  • +

    application/merge-patch+json

    +
    • +
  • +

    application/strategic-merge-patch+json

    +
    • +
+
+
+
+

Produces

+
+
    +
  • +

    application/json

    +
    • +
  • +

    application/yaml

    +
    • +
  • +

    application/vnd.kubernetes.protobuf

    +
    • +
+
+
+
+

Tags

+
+
    +
  • +

    apisfederationv1beta1

    +
    • +
+
+
+
+
+

replace status of the specified Cluster

+
+
+
PUT /apis/federation/v1beta1/clusters/{name}/status
+
+
+
+

Parameters

+ ++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeNameDescriptionRequiredSchemaDefault

QueryParameter

pretty

If true, then the output is pretty printed.

false

string

BodyParameter

body

true

v1beta1.Cluster

PathParameter

name

name of the Cluster

true

string

+ +
+
+

Responses

+ +++++ + + + + + + + + + + + + + + + + + + + +
HTTP CodeDescriptionSchema

200

success

v1beta1.Cluster

201

Created

v1beta1.Cluster

+ +
+
+

Consumes

+
+
    +
  • +

    /

    +
    • +
+
+
+
+

Produces

+
+
    +
  • +

    application/json

    +
    • +
  • +

    application/yaml

    +
    • +
  • +

    application/vnd.kubernetes.protobuf

    +
    • +
+
+
+
+

Tags

+
+
    +
  • +

    apisfederationv1beta1

    +
    • +
+
+
+
+
+

watch individual changes to a list of Cluster

+
+
+
GET /apis/federation/v1beta1/watch/clusters
+
+
+
+

Parameters

+ ++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeNameDescriptionRequiredSchemaDefault

QueryParameter

pretty

If true, then the output is pretty printed.

false

string

QueryParameter

labelSelector

A selector to restrict the list of returned objects by their labels. Defaults to everything.

false

string

QueryParameter

fieldSelector

A selector to restrict the list of returned objects by their fields. Defaults to everything.

false

string

QueryParameter

includeUninitialized

If true, partially initialized resources are included in the response.

false

boolean

QueryParameter

watch

Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.

false

boolean

QueryParameter

resourceVersion

When specified with a watch call, shows changes that occur after that particular version of a resource. Defaults to changes from the beginning of history. When specified for list: - if unset, then the result is returned from remote storage based on quorum-read flag; - if it’s 0, then we simply return what we currently have in cache, no guarantee; - if set to non zero, then the result is at least as fresh as given rv.

false

string

QueryParameter

timeoutSeconds

Timeout for the list/watch call.

false

integer (int32)

QueryParameter

limit

limit is a maximum number of responses to return for a list call. If more items exist, the server will set the continue field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true. +

The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.

false

integer (int32)

QueryParameter

continue

The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server the server will respond with a 410 ResourceExpired error indicating the client must restart their list without the continue field. This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.

false

string

+ +
+
+

Responses

+ +++++ + + + + + + + + + + + + + + +
HTTP CodeDescriptionSchema

200

success

v1.WatchEvent

+ +
+
+

Consumes

+
+
    +
  • +

    /

    +
    • +
+
+
+
+

Produces

+
+
    +
  • +

    application/json

    +
    • +
  • +

    application/yaml

    +
    • +
  • +

    application/vnd.kubernetes.protobuf

    +
    • +
  • +

    application/json;stream=watch

    +
    • +
  • +

    application/vnd.kubernetes.protobuf;stream=watch

    +
    • +
+
+
+
+

Tags

+
+
    +
  • +

    apisfederationv1beta1

    +
    • +
+
+
+
+
+

watch changes to an object of kind Cluster

+
+
+
GET /apis/federation/v1beta1/watch/clusters/{name}
+
+
+
+

Parameters

+ ++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeNameDescriptionRequiredSchemaDefault

QueryParameter

pretty

If true, then the output is pretty printed.

false

string

QueryParameter

labelSelector

A selector to restrict the list of returned objects by their labels. Defaults to everything.

false

string

QueryParameter

fieldSelector

A selector to restrict the list of returned objects by their fields. Defaults to everything.

false

string

QueryParameter

includeUninitialized

If true, partially initialized resources are included in the response.

false

boolean

QueryParameter

watch

Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.

false

boolean

QueryParameter

resourceVersion

When specified with a watch call, shows changes that occur after that particular version of a resource. Defaults to changes from the beginning of history. When specified for list: - if unset, then the result is returned from remote storage based on quorum-read flag; - if it’s 0, then we simply return what we currently have in cache, no guarantee; - if set to non zero, then the result is at least as fresh as given rv.

false

string

QueryParameter

timeoutSeconds

Timeout for the list/watch call.

false

integer (int32)

QueryParameter

limit

limit is a maximum number of responses to return for a list call. If more items exist, the server will set the continue field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true. +

The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.

false

integer (int32)

QueryParameter

continue

The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server the server will respond with a 410 ResourceExpired error indicating the client must restart their list without the continue field. This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.

false

string

PathParameter

name

name of the Cluster

true

string

+ +
+
+

Responses

+ +++++ + + + + + + + + + + + + + + +
HTTP CodeDescriptionSchema

200

success

v1.WatchEvent

+ +
+
+

Consumes

+
+
    +
  • +

    /

    +
    • +
+
+
+
+

Produces

+
+
    +
  • +

    application/json

    +
    • +
  • +

    application/yaml

    +
    • +
  • +

    application/vnd.kubernetes.protobuf

    +
    • +
  • +

    application/json;stream=watch

    +
    • +
  • +

    application/vnd.kubernetes.protobuf;stream=watch

    +
    • +
+
+
+
+

Tags

+
+
    +
  • +

    apisfederationv1beta1

    +
    • +
+
+
+
+
+
+
+ + + \ No newline at end of file diff --git a/content/en/docs/reference/federation/v1/definitions.html b/content/en/docs/reference/federation/v1/definitions.html index 9cfb9cf1d3..b0b55365a7 100755 --- a/content/en/docs/reference/federation/v1/definitions.html +++ b/content/en/docs/reference/federation/v1/definitions.html @@ -1871,7 +1871,7 @@ span.icon > [class^="icon-"], span.icon > [class*=" icon-"] { cursor: default; }

nodePort

-

The port on each node on which this service is exposed when type=NodePort or LoadBalancer. Usually assigned by the system. If specified, it will be allocated to the service if unused or else creation of the service will fail. Default is to auto-allocate a port if the ServiceType of this Service requires one. More info: https://kubernetes.io/docs/concepts/services-networking/service/#nodeport

+

The port on each node on which this service is exposed when type=NodePort or LoadBalancer. Usually assigned by the system. If specified, it will be allocated to the service if unused or else creation of the service will fail. Default is to auto-allocate a port if the ServiceType of this Service requires one. More info: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport

false

integer (int32)

@@ -2350,7 +2350,7 @@ When an object is created, the system will populate this list with the current s

loadBalancerIP

-

Only applies to Service Type: LoadBalancer will get created with the IP specified in this field. This feature depends on whether the underlying cloud-provider supports specifying the loadBalancerIP when a load balancer is created. This field will be ignored if the cloud-provider does not support the feature.

+

Only applies to Service Type: LoadBalancer LoadBalancer will get created with the IP specified in this field. This feature depends on whether the underlying cloud-provider supports specifying the loadBalancerIP when a load balancer is created. This field will be ignored if the cloud-provider does not support the feature.

false

string

@@ -2547,4 +2547,4 @@ Examples:
- + \ No newline at end of file diff --git a/content/en/docs/reference/federation/v1/operations.html b/content/en/docs/reference/federation/v1/operations.html index 98a18c2278..bcca194ab8 100755 --- a/content/en/docs/reference/federation/v1/operations.html +++ b/content/en/docs/reference/federation/v1/operations.html @@ -9526,4 +9526,4 @@ span.icon > [class^="icon-"], span.icon > [class*=" icon-"] { cursor: default; } - + \ No newline at end of file diff --git a/content/en/docs/reference/kubectl/kubectl.md b/content/en/docs/reference/kubectl/kubectl.md index 3d6f5808c2..d1851a3e3e 100755 --- a/content/en/docs/reference/kubectl/kubectl.md +++ b/content/en/docs/reference/kubectl/kubectl.md @@ -9,7 +9,7 @@ kubectl controls the Kubernetes cluster manager ### Synopsis -kubectl controls the Kubernetes cluster manager. +kubectl controls the Kubernetes cluster manager. Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/ @@ -26,13 +26,6 @@ kubectl [flags] - - --allow-verification-with-non-compliant-keys - - - Allow a SignatureVerifier to use keys which are technically non-compliant with RFC6962. - - --alsologtostderr @@ -62,7 +55,7 @@ kubectl [flags] - --cache-dir string     Default: "/Users/zarnold/.kube/http-cache" + --cache-dir string     Default: "/Users/tim/.kube/http-cache" Default HTTP cache directory @@ -159,6 +152,13 @@ kubectl [flags] If non-empty, write log files in this directory + + --log-file string + + + If non-empty, use this log file + + --log-flush-frequency duration     Default: 5s @@ -187,6 +187,20 @@ kubectl [flags] If present, the namespace scope for this CLI request + + --profile string     Default: "none" + + + Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex) + + + + --profile-output string     Default: "profile.pprof" + + + Name of the file to write the profile to + + --request-timeout string     Default: "0" @@ -201,6 +215,13 @@ kubectl [flags] The address and port of the Kubernetes API server + + --skip-headers + + + If true, avoid header prefixes in the log messages + + --stderrthreshold severity     Default: 2 @@ -249,7 +270,6 @@ kubectl [flags] ### SEE ALSO -* [kubectl alpha](/docs/reference/generated/kubectl/kubectl-commands#alpha) - Commands for features in alpha * [kubectl annotate](/docs/reference/generated/kubectl/kubectl-commands#annotate) - Update the annotations on a resource * [kubectl api-resources](/docs/reference/generated/kubectl/kubectl-commands#api-resources) - Print the supported API resources on the server * [kubectl api-versions](/docs/reference/generated/kubectl/kubectl-commands#api-versions) - Print the supported API versions on the server, in the form of "group/version" @@ -267,6 +287,7 @@ kubectl [flags] * [kubectl create](/docs/reference/generated/kubectl/kubectl-commands#create) - Create a resource from a file or from stdin. * [kubectl delete](/docs/reference/generated/kubectl/kubectl-commands#delete) - Delete resources by filenames, stdin, resources and names, or by resources and label selector * [kubectl describe](/docs/reference/generated/kubectl/kubectl-commands#describe) - Show details of a specific resource or group of resources +* [kubectl diff](/docs/reference/generated/kubectl/kubectl-commands#diff) - Diff live version against would-be applied version * [kubectl drain](/docs/reference/generated/kubectl/kubectl-commands#drain) - Drain node in preparation for maintenance * [kubectl edit](/docs/reference/generated/kubectl/kubectl-commands#edit) - Edit a resource on the server * [kubectl exec](/docs/reference/generated/kubectl/kubectl-commands#exec) - Execute a command in a container diff --git a/content/en/docs/reference/kubectl/overview.md b/content/en/docs/reference/kubectl/overview.md index 25388fdf80..bdca0e4840 100644 --- a/content/en/docs/reference/kubectl/overview.md +++ b/content/en/docs/reference/kubectl/overview.md @@ -38,17 +38,17 @@ where `command`, `TYPE`, `NAME`, and `flags` are: * `NAME`: Specifies the name of the resource. Names are case-sensitive. If the name is omitted, details for all resources are displayed, for example `$ kubectl get pods`. When performing an operation on multiple resources, you can specify each resource by type and name or specify one or more files: - + * To specify resources by type and name: - + * To group resources if they are all the same type: `TYPE1 name1 name2 name<#>`.
Example: `$ kubectl get pod example-pod1 example-pod2` - + * To specify multiple resource types individually: `TYPE1/name1 TYPE1/name2 TYPE2/name3 TYPE<#>/name<#>`.
Example: `$ kubectl get pod/example-pod1 replicationcontroller/example-rc1` - + * To specify resources with one or more files: `-f file1 -f file2 -f file<#>` - + * [Use YAML rather than JSON](/docs/concepts/configuration/overview/#general-config-tips) since YAML tends to be more user-friendly, especially for configuration files.
Example: `$ kubectl get pod -f ./pod.yaml` @@ -76,6 +76,7 @@ Operation | Syntax | Description `create` | `kubectl create -f FILENAME [flags]` | Create one or more resources from a file or stdin. `delete` | `kubectl delete (-f FILENAME \| TYPE [NAME \| /NAME \| -l label \| --all]) [flags]` | Delete resources either from a file, stdin, or specifying label selectors, names, resource selectors, or resources. `describe` | `kubectl describe (-f FILENAME \| TYPE [NAME_PREFIX \| /NAME \| -l label]) [flags]` | Display the detailed state of one or more resources. +`diff` | `kubectl diff -f FILENAME [flags]`| Diff file or stdin against live configuration (**BETA**) `edit` | `kubectl edit (-f FILENAME \| TYPE NAME \| TYPE/NAME) [flags]` | Edit and update the definition of one or more resources on the server by using the default editor. `exec` | `kubectl exec POD [-c CONTAINER] [-i] [-t] [flags] [-- COMMAND [args...]]` | Execute a command against a container in a pod. `explain` | `kubectl explain [--include-extended-apis=true] [--recursive=false] [flags]` | Get documentation of various resources. For instance pods, nodes, services, etc. diff --git a/content/en/docs/reference/kubernetes-api/index.md b/content/en/docs/reference/kubernetes-api/index.md index bb5a42fee9..c1868a54ea 100644 --- a/content/en/docs/reference/kubernetes-api/index.md +++ b/content/en/docs/reference/kubernetes-api/index.md @@ -1,5 +1,5 @@ --- -title: v1.12 +title: v1.13 --- -[Kubernetes API v1.12](/docs/reference/generated/kubernetes-api/v1.12/) +[Kubernetes API v1.13](/docs/reference/generated/kubernetes-api/v1.13/) diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm.md index 66869de837..3d11478167 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm.md @@ -5,32 +5,31 @@ kubeadm: easily bootstrap a secure Kubernetes cluster -kubeadm: easily bootstrap a secure Kubernetes cluster. ┌──────────────────────────────────────────────────────────┐ - │ KUBEADM IS CURRENTLY IN BETA │ + │ KUBEADM │ + │ Easily bootstrap a secure Kubernetes cluster │ │ │ - │ But please, try it out and give us feedback at: │ + │ Please give us feedback at: │ │ https://github.com/kubernetes/kubeadm/issues │ - │ and at-mention @kubernetes/sig-cluster-lifecycle-bugs │ - │ or @kubernetes/sig-cluster-lifecycle-feature-requests │ └──────────────────────────────────────────────────────────┘ Example usage: - Create a two-machine cluster with one master (which controls the cluster), - and one node (where your workloads, like Pods and Deployments run). + Create a two-machine cluster with one control-plane node + (which controls the cluster), and one worker node + (where your workloads, like Pods and Deployments run). ┌──────────────────────────────────────────────────────────┐ │ On the first machine: │ ├──────────────────────────────────────────────────────────┤ - │ master# kubeadm init │ + │ control-plane# kubeadm init │ └──────────────────────────────────────────────────────────┘ ┌──────────────────────────────────────────────────────────┐ │ On the second machine: │ ├──────────────────────────────────────────────────────────┤ - │ node# kubeadm join │ + │ worker# kubeadm join │ └──────────────────────────────────────────────────────────┘ You can then repeat the second step on as many other machines as you like. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha.md index 9022c61dbc..ced9d4b528 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha.md @@ -1,10 +1,10 @@ -Experimental sub-commands not yet fully functional. +Kubeadm experimental sub-commands ### Synopsis -Experimental sub-commands not yet fully functional. +Kubeadm experimental sub-commands ### Options diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs.md new file mode 100644 index 0000000000..8548203f74 --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs.md @@ -0,0 +1,50 @@ + +Commands related to handling kubernetes certificates + +### Synopsis + + +Commands related to handling kubernetes certificates + +### Options + + + + + + + + + + + + + + + + +
-h, --help
help for certs
+ + + +### Options inherited from parent commands + + + + + + + + + + + + + + + + +
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
+ + + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew.md similarity index 95% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew.md index 348c268eed..f212b9e12b 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew.md @@ -7,7 +7,7 @@ Renews certificates for a Kubernetes cluster This command is not meant to be run on its own. See list of available subcommands. ``` -kubeadm alpha phase certs renew [flags] +kubeadm alpha certs renew [flags] ``` ### Options diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_all.md similarity index 77% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_all.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_all.md index c5fd726b3c..60fa27fb56 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_all.md @@ -7,7 +7,7 @@ renew all available certificates Renews all known certificates necessary to run the control plan. Renewals are run unconditionally, regardless of expiration date. Renewals can also be run individually for more control. ``` -kubeadm alpha phase certs renew all [flags] +kubeadm alpha certs renew all [flags] ``` ### Options @@ -30,7 +30,21 @@ kubeadm alpha phase certs renew all [flags] --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to a kubeadm configuration file. + + + + --csr-dir string + + + The path to output the CSRs and private keys to + + + + --csr-only + + + Create CSRs instead of generating certificates @@ -44,7 +58,7 @@ kubeadm alpha phase certs renew all [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_apiserver-etcd-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver-etcd-client.md similarity index 78% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_apiserver-etcd-client.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver-etcd-client.md index 255b18b0d1..3489a9d677 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_apiserver-etcd-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver-etcd-client.md @@ -9,7 +9,7 @@ Renews the client apiserver uses to access etcd, and saves them into apiserver-e Extra attributes such as SANs will be based on the existing certificates, there is no need to resupply them. ``` -kubeadm alpha phase certs renew apiserver-etcd-client [flags] +kubeadm alpha certs renew apiserver-etcd-client [flags] ``` ### Options @@ -32,7 +32,21 @@ kubeadm alpha phase certs renew apiserver-etcd-client [flags] --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to a kubeadm configuration file. + + + + --csr-dir string + + + The path to output the CSRs and private keys to + + + + --csr-only + + + Create CSRs instead of generating certificates @@ -46,7 +60,7 @@ kubeadm alpha phase certs renew apiserver-etcd-client [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_apiserver-kubelet-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver-kubelet-client.md similarity index 78% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_apiserver-kubelet-client.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver-kubelet-client.md index 1fc536f2ac..b6e5f7406f 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_apiserver-kubelet-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver-kubelet-client.md @@ -9,7 +9,7 @@ Renews the Client certificate for the API server to connect to kubelet, and save Extra attributes such as SANs will be based on the existing certificates, there is no need to resupply them. ``` -kubeadm alpha phase certs renew apiserver-kubelet-client [flags] +kubeadm alpha certs renew apiserver-kubelet-client [flags] ``` ### Options @@ -32,7 +32,21 @@ kubeadm alpha phase certs renew apiserver-kubelet-client [flags] --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to a kubeadm configuration file. + + + + --csr-dir string + + + The path to output the CSRs and private keys to + + + + --csr-only + + + Create CSRs instead of generating certificates @@ -46,7 +60,7 @@ kubeadm alpha phase certs renew apiserver-kubelet-client [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_apiserver.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver.md similarity index 73% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_apiserver.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver.md index c56b907b99..d45c3ca73a 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_apiserver.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver.md @@ -1,15 +1,15 @@ -Generates the certificate for serving the kubernetes API +Generates the certificate for serving the Kubernetes API ### Synopsis -Renews the certificate for serving the kubernetes API, and saves them into apiserver.cert and apiserver.key files. +Renews the certificate for serving the Kubernetes API, and saves them into apiserver.cert and apiserver.key files. Extra attributes such as SANs will be based on the existing certificates, there is no need to resupply them. ``` -kubeadm alpha phase certs renew apiserver [flags] +kubeadm alpha certs renew apiserver [flags] ``` ### Options @@ -32,7 +32,21 @@ kubeadm alpha phase certs renew apiserver [flags] --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to a kubeadm configuration file. + + + + --csr-dir string + + + The path to output the CSRs and private keys to + + + + --csr-only + + + Create CSRs instead of generating certificates @@ -46,7 +60,7 @@ kubeadm alpha phase certs renew apiserver [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_etcd-healthcheck-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-healthcheck-client.md similarity index 78% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_etcd-healthcheck-client.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-healthcheck-client.md index 1ecafc8366..3c798047f3 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_etcd-healthcheck-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-healthcheck-client.md @@ -9,7 +9,7 @@ Renews the client certificate for liveness probes to healtcheck etcd, and saves Extra attributes such as SANs will be based on the existing certificates, there is no need to resupply them. ``` -kubeadm alpha phase certs renew etcd-healthcheck-client [flags] +kubeadm alpha certs renew etcd-healthcheck-client [flags] ``` ### Options @@ -32,7 +32,21 @@ kubeadm alpha phase certs renew etcd-healthcheck-client [flags] --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to a kubeadm configuration file. + + + + --csr-dir string + + + The path to output the CSRs and private keys to + + + + --csr-only + + + Create CSRs instead of generating certificates @@ -46,7 +60,7 @@ kubeadm alpha phase certs renew etcd-healthcheck-client [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_etcd-peer.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-peer.md similarity index 78% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_etcd-peer.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-peer.md index 75da17a311..b8cf86bad1 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_etcd-peer.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-peer.md @@ -9,7 +9,7 @@ Renews the credentials for etcd nodes to communicate with each other, and saves Extra attributes such as SANs will be based on the existing certificates, there is no need to resupply them. ``` -kubeadm alpha phase certs renew etcd-peer [flags] +kubeadm alpha certs renew etcd-peer [flags] ``` ### Options @@ -32,7 +32,21 @@ kubeadm alpha phase certs renew etcd-peer [flags] --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to a kubeadm configuration file. + + + + --csr-dir string + + + The path to output the CSRs and private keys to + + + + --csr-only + + + Create CSRs instead of generating certificates @@ -46,7 +60,7 @@ kubeadm alpha phase certs renew etcd-peer [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_etcd-server.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-server.md similarity index 78% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_etcd-server.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-server.md index a44a852aeb..123111cd55 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_etcd-server.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-server.md @@ -9,7 +9,7 @@ Renews the certificate for serving etcd, and saves them into etcd/server.cert an Extra attributes such as SANs will be based on the existing certificates, there is no need to resupply them. ``` -kubeadm alpha phase certs renew etcd-server [flags] +kubeadm alpha certs renew etcd-server [flags] ``` ### Options @@ -32,7 +32,21 @@ kubeadm alpha phase certs renew etcd-server [flags] --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to a kubeadm configuration file. + + + + --csr-dir string + + + The path to output the CSRs and private keys to + + + + --csr-only + + + Create CSRs instead of generating certificates @@ -46,7 +60,7 @@ kubeadm alpha phase certs renew etcd-server [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_front-proxy-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_front-proxy-client.md similarity index 78% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_front-proxy-client.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_front-proxy-client.md index 41ad8887a1..ed9f18495a 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_renew_front-proxy-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_front-proxy-client.md @@ -9,7 +9,7 @@ Renews the client for the front proxy, and saves them into front-proxy-client.ce Extra attributes such as SANs will be based on the existing certificates, there is no need to resupply them. ``` -kubeadm alpha phase certs renew front-proxy-client [flags] +kubeadm alpha certs renew front-proxy-client [flags] ``` ### Options @@ -32,7 +32,21 @@ kubeadm alpha phase certs renew front-proxy-client [flags] --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to a kubeadm configuration file. + + + + --csr-dir string + + + The path to output the CSRs and private keys to + + + + --csr-only + + + Create CSRs instead of generating certificates @@ -46,7 +60,7 @@ kubeadm alpha phase certs renew front-proxy-client [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubeconfig.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubeconfig.md new file mode 100644 index 0000000000..24b0f4d901 --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubeconfig.md @@ -0,0 +1,52 @@ + +Kubeconfig file utilities + +### Synopsis + + +Kubeconfig file utilities. + + Alpha Disclaimer: this command is currently alpha. + +### Options + + + + + + + + + + + + + + + + +
-h, --help
help for kubeconfig
+ + + +### Options inherited from parent commands + + + + + + + + + + + + + + + + +
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
+ + + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_user.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubeconfig_user.md similarity index 85% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_user.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubeconfig_user.md index 9f7c0d92f0..c12f50d3cf 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_user.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubeconfig_user.md @@ -6,17 +6,17 @@ Outputs a kubeconfig file for an additional user Outputs a kubeconfig file for an additional user. -Alpha Disclaimer: this command is currently alpha. + Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase kubeconfig user [flags] +kubeadm alpha kubeconfig user [flags] ``` ### Examples ``` # Outputs a kubeconfig file for an additional user named foo - kubeadm alpha phase kubeconfig user --client-name=foo + kubeadm alpha kubeconfig user --client-name=foo ``` ### Options @@ -63,13 +63,6 @@ kubeadm alpha phase kubeconfig user [flags] help for user - - --kubeconfig-dir string     Default: "/etc/kubernetes" - - - The path where to save the kubeconfig file - - --org stringSlice diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet.md similarity index 95% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet.md index 648b219e76..5479cf594e 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet.md @@ -1,5 +1,5 @@ -Commands related to handling the kubelet. +Commands related to handling the kubelet ### Synopsis diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config.md similarity index 96% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config.md index 2c2d8ca004..be9d4fcc26 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config.md @@ -1,5 +1,5 @@ -Handles kubelet configuration. +Utilities for kubelet configuration ### Synopsis diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config_download.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config_download.md similarity index 95% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config_download.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config_download.md index 8f1118ee1e..20ad0bbf4c 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config_download.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config_download.md @@ -9,7 +9,7 @@ Downloads the kubelet configuration from a ConfigMap of the form "kubelet-config Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase kubelet config download [flags] +kubeadm alpha kubelet config download [flags] ``` ### Examples @@ -42,7 +42,7 @@ kubeadm alpha phase kubelet config download [flags] --kubeconfig string     Default: "/etc/kubernetes/kubelet.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config_enable-dynamic.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config_enable-dynamic.md similarity index 95% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config_enable-dynamic.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config_enable-dynamic.md index 91bf90d15e..fdad69928b 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config_enable-dynamic.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config_enable-dynamic.md @@ -11,7 +11,7 @@ WARNING: This feature is still experimental, and disabled by default. Enable onl Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase kubelet config enable-dynamic [flags] +kubeadm alpha kubelet config enable-dynamic [flags] ``` ### Examples @@ -44,7 +44,7 @@ kubeadm alpha phase kubelet config enable-dynamic [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_all.md deleted file mode 100644 index 0169c68221..0000000000 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_all.md +++ /dev/null @@ -1,122 +0,0 @@ - -Makes all the bootstrap token configurations and creates an initial token - -### Synopsis - - -Bootstrap tokens are used for establishing bidirectional trust between a node joining the cluster and a the master node. - -This command makes all the configurations required to make bootstrap tokens works and then creates an initial token. - -Alpha Disclaimer: this command is currently alpha. - -``` -kubeadm alpha phase bootstrap-token all [flags] -``` - -### Examples - -``` - # Makes all the bootstrap token configurations and creates an initial token, functionally - # equivalent to what generated by kubeadm init. - kubeadm alpha phase bootstrap-token all -``` - -### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental
--description string
A human friendly description of how this token is used.
--groups stringSlice     Default: [system:bootstrappers:kubeadm:default-node-token]
Extra groups that this token will authenticate as when used for authentication. Must match "\\Asystem:bootstrappers:[a-z0-9:-]{0,255}[a-z0-9]\\z"
-h, --help
help for all
--skip-token-print
Skip printing of the bootstrap token
--token string
The token to use for establishing bidirectional trust between nodes and masters. The format is [a-z0-9]{6}\.[a-z0-9]{16} - e.g. abcdef.0123456789abcdef
--token-ttl duration     Default: 24h0m0s
The duration before the token is automatically deleted (e.g. 1s, 2m, 3h). If set to '0', the token will never expire
--usages stringSlice     Default: [signing,authentication]
Describes the ways in which this token can be used. You can pass --usages multiple times or provide a comma separated list of options. Valid options: [signing,authentication]
- - - -### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - - -
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_cluster-info.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_cluster-info.md deleted file mode 100644 index 5cc7f3fc05..0000000000 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_cluster-info.md +++ /dev/null @@ -1,65 +0,0 @@ - -Uploads the cluster-info ConfigMap from the given kubeconfig file - -### Synopsis - - -Uploads the "cluster-info" ConfigMap in the "kube-public" namespace, populating it with cluster information extracted from the given kubeconfig file. The ConfigMap is used for the node bootstrap process in its initial phases, before the client trusts the API server. - -See online documentation about Authenticating with Bootstrap Tokens for more details. - -Alpha Disclaimer: this command is currently alpha. - -``` -kubeadm alpha phase bootstrap-token cluster-info [flags] -``` - -### Options - - - - - - - - - - - - - - - - -
-h, --help
help for cluster-info
- - - -### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - - -
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_create.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_create.md deleted file mode 100644 index aed484bb38..0000000000 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_create.md +++ /dev/null @@ -1,114 +0,0 @@ - -Creates a bootstrap token to be used for node joining - -### Synopsis - - -Creates a bootstrap token. If no token value is given, kubeadm will generate a random token instead. - -Alternatively, you can use kubeadm token. - -Alpha Disclaimer: this command is currently alpha. - -``` -kubeadm alpha phase bootstrap-token create [flags] -``` - -### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental
--description string
A human friendly description of how this token is used.
--groups stringSlice     Default: [system:bootstrappers:kubeadm:default-node-token]
Extra groups that this token will authenticate as when used for authentication. Must match "\\Asystem:bootstrappers:[a-z0-9:-]{0,255}[a-z0-9]\\z"
-h, --help
help for create
--skip-token-print
Skip printing of the bootstrap token
--token string
The token to use for establishing bidirectional trust between nodes and masters. The format is [a-z0-9]{6}\.[a-z0-9]{16} - e.g. abcdef.0123456789abcdef
--token-ttl duration     Default: 24h0m0s
The duration before the token is automatically deleted (e.g. 1s, 2m, 3h). If set to '0', the token will never expire
--usages stringSlice     Default: [signing,authentication]
Describes the ways in which this token can be used. You can pass --usages multiple times or provide a comma separated list of options. Valid options: [signing,authentication]
- - - -### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - - -
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_node_allow-auto-approve.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_node_allow-auto-approve.md deleted file mode 100644 index 405ece9bce..0000000000 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_node_allow-auto-approve.md +++ /dev/null @@ -1,65 +0,0 @@ - -Configures RBAC rules to allow the csrapprover controller automatically approve CSRs from a node bootstrap token - -### Synopsis - - -Configures RBAC rules to allow the csrapprover controller to automatically approve certificate signing requests generated by nodes joining the cluster. It configures also RBAC rules for certificates rotation (with auto approval of new certificates). - -See online documentation about TLS bootstrapping for more details. - -Alpha Disclaimer: this command is currently alpha. - -``` -kubeadm alpha phase bootstrap-token node allow-auto-approve [flags] -``` - -### Options - - - - - - - - - - - - - - - - -
-h, --help
help for allow-auto-approve
- - - -### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - - -
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_node_allow-post-csrs.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_node_allow-post-csrs.md deleted file mode 100644 index 8aee98f2ec..0000000000 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_node_allow-post-csrs.md +++ /dev/null @@ -1,65 +0,0 @@ - -Configures RBAC to allow node bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials - -### Synopsis - - -Configures RBAC rules to allow node bootstrap tokens to post a certificate signing request, thus enabling nodes joining the cluster to request long term certificate credentials. - -See online documentation about TLS bootstrapping for more details. - -Alpha Disclaimer: this command is currently alpha. - -``` -kubeadm alpha phase bootstrap-token node allow-post-csrs [flags] -``` - -### Options - - - - - - - - - - - - - - - - -
-h, --help
help for allow-post-csrs
- - - -### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - - -
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_all.md deleted file mode 100644 index 07916e76e8..0000000000 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_all.md +++ /dev/null @@ -1,111 +0,0 @@ - -Generates all PKI assets necessary to establish the control plane - -### Synopsis - - -Generates a self-signed CA to provision identities for each component in the cluster (including nodes) and client certificates to be used by various components. - -If a given certificate and private key pair both exist, kubeadm skips the generation step and existing files will be used. - -Alpha Disclaimer: this command is currently alpha. - -``` -kubeadm alpha phase certs all [flags] -``` - -### Examples - -``` - # Creates all PKI assets necessary to establish the control plane, - # functionally equivalent to what generated by kubeadm init. - kubeadm alpha phase certs all - - # Creates all PKI assets using options read from a configuration file. - kubeadm alpha phase certs all --config masterconfiguration.yaml -``` - -### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
The IP address the API server is accessible on, to use for the API server serving cert
--apiserver-cert-extra-sans stringSlice
Optional extra altnames to use for the API server serving cert. Can be both IP addresses and DNS names
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save the certificates
--config string
Path to kubeadm config file (WARNING: Usage of a configuration file is experimental)
-h, --help
help for all
--service-cidr string     Default: "10.96.0.0/12"
Alternative range of IP address for service VIPs, from which derives the internal API server VIP that will be added to the API Server serving cert
--service-dns-domain string     Default: "cluster.local"
Alternative domain for services, to use for the API server serving cert
- - - -### Options inherited from parent commands - - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_all.md deleted file mode 100644 index ffb3610e9b..0000000000 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_all.md +++ /dev/null @@ -1,109 +0,0 @@ - -Generates all kubeconfig files necessary to establish the control plane and the admin kubeconfig file - -### Synopsis - - -Generates all kubeconfig files necessary to establish the control plane and the admin kubeconfig file. - -Alpha Disclaimer: this command is currently alpha. - -``` -kubeadm alpha phase kubeconfig all [flags] -``` - -### Examples - -``` - # Generates all kubeconfig files, functionally equivalent to what generated - # by kubeadm init. - kubeadm alpha phase kubeconfig all - - # Generates all kubeconfig files using options read from a configuration file. - kubeadm alpha phase kubeconfig all --config masterconfiguration.yaml -``` - -### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
The IP address the API server is accessible on
--apiserver-bind-port int32     Default: 6443
The port the API server is accessible on
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where certificates are stored
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental
-h, --help
help for all
--kubeconfig-dir string     Default: "/etc/kubernetes"
The path where to save the kubeconfig file
--node-name string
The node name that should be used for the kubelet client certificate
- - - -### Options inherited from parent commands - - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config_write-to-disk.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config_write-to-disk.md deleted file mode 100644 index 02eaee18e0..0000000000 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config_write-to-disk.md +++ /dev/null @@ -1,70 +0,0 @@ - -Writes kubelet configuration to disk, either based on the --config argument. - -### Synopsis - - -Writes kubelet configuration to disk, based on the kubeadm configuration passed via "--config". - -Alpha Disclaimer: this command is currently alpha. - -``` -kubeadm alpha phase kubelet config write-to-disk [flags] -``` - -### Examples - -``` - # Extracts the kubelet configuration from a kubeadm configuration file - kubeadm alpha phase kubelet config write-to-disk --config kubeadm.yaml -``` - -### Options - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to kubeadm config file (WARNING: Usage of a configuration file is experimental)
-h, --help
help for write-to-disk
- - - -### Options inherited from parent commands - - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_write-env-file.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_write-env-file.md deleted file mode 100644 index eadafa78dd..0000000000 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_write-env-file.md +++ /dev/null @@ -1,73 +0,0 @@ - -Writes an environment file with runtime flags for the kubelet. - -### Synopsis - - -Writes an environment file with flags that should be passed to the kubelet executing on the master or node. This --config flag can either consume a InitConfiguration object or a JoinConfiguration one, as this function is used for both "kubeadm init" and "kubeadm join". - -Alpha Disclaimer: this command is currently alpha. - -``` -kubeadm alpha phase kubelet write-env-file [flags] -``` - -### Examples - -``` - # Writes a dynamic environment file with kubelet flags from a InitConfiguration file. - kubeadm alpha phase kubelet write-env-file --config masterconfig.yaml - - # Writes a dynamic environment file with kubelet flags from a JoinConfiguration file. - kubeadm alpha phase kubelet write-env-file --config nodeconfig.yaml -``` - -### Options - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to kubeadm config file (WARNING: Usage of a configuration file is experimental)
-h, --help
help for write-env-file
- - - -### Options inherited from parent commands - - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_preflight.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_preflight.md deleted file mode 100644 index b41e864631..0000000000 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_preflight.md +++ /dev/null @@ -1,64 +0,0 @@ - -Run pre-flight checks - -### Synopsis - - -This command is not meant to be run on its own. See list of available subcommands. - -### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to kubeadm config file (WARNING: Usage of a configuration file is experimental)
-h, --help
help for preflight
--ignore-preflight-errors stringSlice
A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.
- - - -### Options inherited from parent commands - - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_etcd.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_preflight.md similarity index 92% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_etcd.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_preflight.md index 60819008a3..d88f71b0d9 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_etcd.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_preflight.md @@ -1,5 +1,5 @@ -Generates static Pod manifest file for etcd. +Commands related to pre-flight checks ### Synopsis @@ -19,7 +19,7 @@ This command is not meant to be run on its own. See list of available subcommand -h, --help - help for etcd + help for preflight diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_preflight_node.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_preflight_node.md similarity index 89% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_preflight_node.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_preflight_node.md index 2c0cfe2247..47be57c832 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_preflight_node.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_preflight_node.md @@ -9,14 +9,14 @@ Run node pre-flight checks, functionally equivalent to what implemented by kubea Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase preflight node [flags] +kubeadm alpha preflight node [flags] ``` ### Examples ``` # Run node pre-flight checks. - kubeadm alpha phase preflight node + kubeadm alpha preflight node ``` ### Options @@ -28,6 +28,13 @@ kubeadm alpha phase preflight node [flags] + + --config string + + + Path to a kubeadm configuration file. + + -h, --help @@ -35,6 +42,13 @@ kubeadm alpha phase preflight node [flags] help for node + + --ignore-preflight-errors stringSlice + + + A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks. + + @@ -49,20 +63,6 @@ kubeadm alpha phase preflight node [flags] - - --config string - - - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) - - - - --ignore-preflight-errors stringSlice - - - A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks. - - --rootfs string diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_selfhosting.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_selfhosting.md similarity index 100% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_selfhosting.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_selfhosting.md diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_selfhosting_convert-from-staticpods.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_selfhosting_pivot.md similarity index 76% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_selfhosting_convert-from-staticpods.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_selfhosting_pivot.md index fc403a9d46..4f04e48126 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_selfhosting_convert-from-staticpods.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_selfhosting_pivot.md @@ -11,17 +11,15 @@ See the documentation for self-hosting limitations. Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase selfhosting convert-from-staticpods [flags] +kubeadm alpha selfhosting pivot [flags] ``` ### Examples ``` - # Converts a static Pod-hosted control plane into a self-hosted one, - # functionally equivalent to what generated by kubeadm init executed - # with --feature-gates=SelfHosting=true. + # Converts a static Pod-hosted control plane into a self-hosted one. - kubeadm alpha phase selfhosting convert-from-staticpods + kubeadm alpha phase self-hosting convert-from-staticpods ``` ### Options @@ -48,24 +46,31 @@ kubeadm alpha phase selfhosting convert-from-staticpods [flags] - --feature-gates string + -f, --force - A set of key=value pairs that describe feature gates for various features. Options are:
Auditing=true|false (ALPHA - default=false)
CoreDNS=true|false (default=true)
DynamicKubeletConfig=true|false (BETA - default=false) + Pivot the cluster without prompting for confirmation -h, --help - help for convert-from-staticpods + help for pivot --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + + + + -s, --store-certs-in-secrets + + + Enable storing certs in secrets diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config.md index b840595320..6c615daacd 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config.md @@ -35,7 +35,7 @@ kubeadm config [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images.md index ee309006b8..4b1073d17b 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images.md @@ -44,7 +44,7 @@ kubeadm config images [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_list.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_list.md index 345d832898..fac2d052f2 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_list.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_list.md @@ -30,7 +30,7 @@ kubeadm config images list [flags] --feature-gates string - A set of key=value pairs that describe feature gates for various features. Options are:
Auditing=true|false (ALPHA - default=false)
CoreDNS=true|false (default=true)
DynamicKubeletConfig=true|false (BETA - default=false) + A set of key=value pairs that describe feature gates for various features. Options are:
@@ -65,7 +65,7 @@ kubeadm config images list [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_pull.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_pull.md index 83b3a6c10c..7cb2173a26 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_pull.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_pull.md @@ -37,7 +37,7 @@ kubeadm config images pull [flags] --feature-gates string - A set of key=value pairs that describe feature gates for various features. Options are:
Auditing=true|false (ALPHA - default=false)
CoreDNS=true|false (default=true)
DynamicKubeletConfig=true|false (BETA - default=false) + A set of key=value pairs that describe feature gates for various features. Options are:
@@ -72,7 +72,7 @@ kubeadm config images pull [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_migrate.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_migrate.md index eff30744e3..6d84dbd526 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_migrate.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_migrate.md @@ -8,10 +8,10 @@ Read an older version of the kubeadm configuration API types from a file, and ou This command lets you convert configuration objects of older versions to the latest supported version, locally in the CLI tool without ever touching anything in the cluster. In this version of kubeadm, the following API versions are supported: -- kubeadm.k8s.io/v1alpha2 - kubeadm.k8s.io/v1alpha3 +- kubeadm.k8s.io/v1beta1 -Further, kubeadm can only write out config of version "kubeadm.k8s.io/v1alpha3", but read both types. +Further, kubeadm can only write out config of version "kubeadm.k8s.io/v1beta1", but read both types. So regardless of what version you pass to the --old-config parameter here, the API object will be read, deserialized, defaulted, converted, validated, and re-serialized when written to stdout or --new-config if specified. @@ -72,7 +72,7 @@ kubeadm config migrate [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_node.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print.md similarity index 83% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_node.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print.md index ef5cab6139..ea98b1fa30 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token_node.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print.md @@ -1,10 +1,14 @@ -Configures the node bootstrap process +Print configuration ### Synopsis -This command is not meant to be run on its own. See list of available subcommands. +This command prints configurations for subcommands provided. + +``` +kubeadm config print [flags] +``` ### Options @@ -19,7 +23,7 @@ This command is not meant to be run on its own. See list of available subcommand -h, --help - help for node + help for print @@ -40,7 +44,7 @@ This command is not meant to be run on its own. See list of available subcommand --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print-default.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_init-defaults.md similarity index 61% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print-default.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_init-defaults.md index ede571108c..afdc7c3d4e 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print-default.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_init-defaults.md @@ -1,19 +1,18 @@ -Print the default values for a kubeadm configuration object. +Print default init configuration, that can be used for 'kubeadm init' ### Synopsis -This command prints the default InitConfiguration object that is used for 'kubeadm init' and 'kubeadm upgrade', -and the default JoinConfiguration object that is used for 'kubeadm join'. +This command prints objects such as the default init configuration that is used for 'kubeadm init'. -Note that sensitive values like the Bootstrap Token fields are replaced with silly values like {"abcdef.0123456789abcdef" "" "nil" [] []} in order to pass validation but +Note that sensitive values like the Bootstrap Token fields are replaced with placeholder values like {"abcdef.0123456789abcdef" "" "nil" [] []} in order to pass validation but not perform the real computation for creating a token. ``` -kubeadm config print-default [flags] +kubeadm config print init-defaults [flags] ``` ### Options @@ -26,17 +25,17 @@ kubeadm config print-default [flags] - --api-objects stringSlice + --component-configs stringSlice - A comma-separated list for API objects to print the default values for. Available values: [InitConfiguration ClusterConfiguration JoinConfiguration KubeProxyConfiguration KubeletConfiguration MasterConfiguration]. This flag unset means 'print all known objects' + A comma-separated list for component config API objects to print the default values for. Available values: [KubeProxyConfiguration KubeletConfiguration]. If this flag is not set, no component configs will be printed. -h, --help - help for print-default + help for init-defaults @@ -57,7 +56,7 @@ kubeadm config print-default [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_upload-config.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_join-defaults.md similarity index 59% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_upload-config.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_join-defaults.md index 657b643004..d26a83dc92 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_upload-config.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_join-defaults.md @@ -1,24 +1,18 @@ -Uploads the currently used configuration for kubeadm to a ConfigMap +Print default join configuration, that can be used for 'kubeadm join' ### Synopsis -Uploads the kubeadm init configuration of your cluster to a ConfigMap called kubeadm-config in the kube-system namespace. This enables correct configuration of system components and a seamless user experience when upgrading. -Alternatively, you can use kubeadm config. +This command prints objects such as the default join configuration that is used for 'kubeadm join'. + +Note that sensitive values like the Bootstrap Token fields are replaced with placeholder values like {"abcdef.0123456789abcdef" "" "nil" [] []} in order to pass validation but +not perform the real computation for creating a token. -Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase upload-config [flags] -``` - -### Examples - -``` - # uploads the configuration of your cluster - kubeadm alpha phase upload-config --config=myConfig.yaml +kubeadm config print join-defaults [flags] ``` ### Options @@ -31,24 +25,17 @@ kubeadm alpha phase upload-config [flags] - --config string + --component-configs stringSlice - Path to a kubeadm config file. WARNING: Usage of a configuration file is experimental + A comma-separated list for component config API objects to print the default values for. Available values: [KubeProxyConfiguration KubeletConfiguration]. If this flag is not set, no component configs will be printed. -h, --help - help for upload-config - - - - --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - - - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + help for join-defaults @@ -65,6 +52,13 @@ kubeadm alpha phase upload-config [flags] + + --kubeconfig string     Default: "/etc/kubernetes/admin.conf" + + + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + + --rootfs string diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload.md index d16f6f9124..b4e825099a 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload.md @@ -44,7 +44,7 @@ kubeadm config upload [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload_from-file.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload_from-file.md index 87764b606f..51d7987479 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload_from-file.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload_from-file.md @@ -57,7 +57,7 @@ kubeadm config upload from-file [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload_from-flags.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload_from-flags.md index bb660cbe41..08c1adb2f1 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload_from-flags.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload_from-flags.md @@ -64,7 +64,7 @@ kubeadm config upload from-flags [flags] --feature-gates string - A set of key=value pairs that describe feature gates for various features. Options are:
Auditing=true|false (ALPHA - default=false)
CoreDNS=true|false (default=true)
DynamicKubeletConfig=true|false (BETA - default=false) + A set of key=value pairs that describe feature gates for various features. Options are:
@@ -127,7 +127,7 @@ kubeadm config upload from-flags [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_view.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_view.md index 664055d815..a389c5d947 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_view.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_view.md @@ -48,7 +48,7 @@ kubeadm config view [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md index c476ed5803..e626212ffd 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md @@ -6,6 +6,44 @@ Run this command in order to set up the Kubernetes master. Run this command in order to set up the Kubernetes master. +The "init" command executes the following phases: +``` +preflight Run master pre-flight checks +kubelet-start Writes kubelet settings and (re)starts the kubelet +certs Certificate generation + /ca Generates the self-signed Kubernetes CA to provision identities for other Kubernetes components + /apiserver Generates the certificate for serving the Kubernetes API + /apiserver-kubelet-client Generates the Client certificate for the API server to connect to kubelet + /front-proxy-ca Generates the self-signed CA to provision identities for front proxy + /front-proxy-client Generates the client for the front proxy + /etcd-ca Generates the self-signed CA to provision identities for etcd + /etcd-server Generates the certificate for serving etcd + /etcd-peer Generates the credentials for etcd nodes to communicate with each other + /etcd-healthcheck-client Generates the client certificate for liveness probes to healtcheck etcd + /apiserver-etcd-client Generates the client apiserver uses to access etcd + /sa Generates a private key for signing service account tokens along with its public key +kubeconfig Generates all kubeconfig files necessary to establish the control plane and the admin kubeconfig file + /admin Generates a kubeconfig file for the admin to use and for kubeadm itself + /kubelet Generates a kubeconfig file for the kubelet to use *only* for cluster bootstrapping purposes + /controller-manager Generates a kubeconfig file for the controller manager to use + /scheduler Generates a kubeconfig file for the scheduler to use +control-plane Generates all static Pod manifest files necessary to establish the control plane + /apiserver Generates the kube-apiserver static Pod manifest + /controller-manager Generates the kube-controller-manager static Pod manifest + /scheduler Generates the kube-scheduler static Pod manifest +etcd Generates static Pod manifest file for local etcd. + /local Generates the static Pod manifest file for a local, single-node local etcd instance. +upload-config Uploads the kubeadm and kubelet configuration to a ConfigMap + /kubeadm Uploads the kubeadm ClusterConfiguration to a ConfigMap + /kubelet Uploads the kubelet component config to a ConfigMap +mark-control-plane Mark a node as a control-plane +bootstrap-token Generates bootstrap tokens used to join a node to a cluster +addon Installs required addons for passing Conformance tests + /coredns Installs the CoreDNS addon to a Kubernetes cluster + /kube-proxy Installs the kube-proxy addon to a Kubernetes cluster +``` + + ``` kubeadm init [flags] ``` @@ -72,7 +110,7 @@ kubeadm init [flags] --feature-gates string - A set of key=value pairs that describe feature gates for various features. Options are:
Auditing=true|false (ALPHA - default=false)
CoreDNS=true|false (default=true)
DynamicKubeletConfig=true|false (BETA - default=false) + A set of key=value pairs that describe feature gates for various features. Options are:
@@ -89,6 +127,13 @@ kubeadm init [flags] A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks. + + --image-repository string     Default: "k8s.gcr.io" + + + Choose a container registry to pull control plane images from + + --kubernetes-version string     Default: "stable-1" @@ -124,6 +169,13 @@ kubeadm init [flags] Use alternative domain for services, e.g. "myorg.internal". + + --skip-phases stringSlice + + + List of phases to be skipped + + --skip-token-print diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase.md new file mode 100644 index 0000000000..b2946caf1f --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase.md @@ -0,0 +1,50 @@ + +use this command to invoke single phase of the init workflow + +### Synopsis + + +use this command to invoke single phase of the init workflow + +### Options + + + + + + + + + + + + + + + + +
-h, --help
help for phase
+ + + +### Options inherited from parent commands + + + + + + + + + + + + + + + + +
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
+ + + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_addon.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon.md similarity index 95% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_addon.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon.md index 834becc463..1ddc02f451 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_addon.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon.md @@ -6,6 +6,10 @@ Installs required addons for passing Conformance tests This command is not meant to be run on its own. See list of available subcommands. +``` +kubeadm init phase addon [flags] +``` + ### Options diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_addon_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_all.md similarity index 70% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_addon_all.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_all.md index f30b47e323..76e1055467 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_addon_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_all.md @@ -1,24 +1,13 @@ -Installs all addons to a Kubernetes cluster +Installs all the addons ### Synopsis -Installs the CoreDNS and the kube-proxy addons components via the API server. Please note that although the DNS server is deployed, it will not be scheduled until CNI is installed. - -Alpha Disclaimer: this command is currently alpha. +Installs all the addons ``` -kubeadm alpha phase addon all [flags] -``` - -### Examples - -``` - # Installs the CoreDNS and the kube-proxy addons components via the API server, - # functionally equivalent to what installed by kubeadm init. - - kubeadm alpha phase selfhosting from-staticpods +kubeadm init phase addon all [flags] ``` ### Options @@ -34,28 +23,28 @@ kubeadm alpha phase addon all [flags] - + - + - + - + @@ -76,35 +65,35 @@ kubeadm alpha phase addon all [flags] - + - + - + - + - + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_addon_coredns.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_coredns.md similarity index 80% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_addon_coredns.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_coredns.md index 4387b42f8e..a358676d53 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_addon_coredns.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_coredns.md @@ -4,12 +4,10 @@ Installs the CoreDNS addon to a Kubernetes cluster ### Synopsis -Installs the CoreDNS addon components via the API server. Please note that although the DNS server is deployed, it will not be scheduled until CNI is installed. - -Alpha Disclaimer: this command is currently alpha. +Installs the CoreDNS addon components via the API server. Please note that although the DNS server is deployed, it will not be scheduled until CNI is installed. ``` -kubeadm alpha phase addon coredns [flags] +kubeadm init phase addon coredns [flags] ``` ### Options @@ -25,14 +23,14 @@ kubeadm alpha phase addon coredns [flags] - + - + @@ -53,28 +51,28 @@ kubeadm alpha phase addon coredns [flags] - + - + - + - + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_addon_kube-proxy.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_kube-proxy.md similarity index 77% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_addon_kube-proxy.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_kube-proxy.md index 0c36be4676..f967ae1647 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_addon_kube-proxy.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_kube-proxy.md @@ -4,12 +4,10 @@ Installs the kube-proxy addon to a Kubernetes cluster ### Synopsis -Installs the kube-proxy addon components via the API server. - -Alpha Disclaimer: this command is currently alpha. +Installs the kube-proxy addon components via the API server. ``` -kubeadm alpha phase addon kube-proxy [flags] +kubeadm init phase addon kube-proxy [flags] ``` ### Options @@ -25,21 +23,21 @@ kubeadm alpha phase addon kube-proxy [flags] - + - + - + @@ -60,21 +58,21 @@ kubeadm alpha phase addon kube-proxy [flags] - + - + - + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_bootstrap-token.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_bootstrap-token.md new file mode 100644 index 0000000000..c853d7d2fb --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_bootstrap-token.md @@ -0,0 +1,85 @@ + +Generates bootstrap tokens used to join a node to a cluster + +### Synopsis + + +Bootstrap tokens are used for establishing bidirectional trust between a node joining the cluster and a the control-plane node. + +This command makes all the configurations required to make bootstrap tokens works and then creates an initial token. + +``` +kubeadm init phase bootstrap-token [flags] +``` + +### Examples + +``` + # Makes all the bootstrap token configurations and creates an initial token, functionally + # equivalent to what generated by kubeadm init. + kubeadm init phase bootstrap-token +``` + +### Options + +
--apiserver-advertise-address string
The IP address the API server is accessible onThe IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
The port the API server is accessible onPort for the API Server to bind to.
--config string
Path to a kubeadm config file. WARNING: Usage of a configuration file is experimentalPath to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--feature-gates string
A set of key=value pairs that describe feature gates for various features. Options are:
Auditing=true|false (ALPHA - default=false)
CoreDNS=true|false (default=true)
DynamicKubeletConfig=true|false (BETA - default=false)		A set of key=value pairs that describe feature gates for various features. Options are:
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control planeChoose a specific Kubernetes version for the control plane.
--pod-network-cidr string
The range of IP addresses used for the Pod networkSpecify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
--service-cidr string     Default: "10.96.0.0/12"
The range of IP address used for service VIPsUse alternative range of IP address for service VIPs.
--service-dns-domain string     Default: "cluster.local"
Alternative domain for servicesUse alternative domain for services, e.g. "myorg.internal".
--config string
Path to a kubeadm config file. WARNING: Usage of a configuration file is experimentalPath to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--feature-gates string
A set of key=value pairs that describe feature gates for various features. Options are:
Auditing=true|false (ALPHA - default=false)
CoreDNS=true|false (default=true)
DynamicKubeletConfig=true|false (BETA - default=false)		A set of key=value pairs that describe feature gates for various features. Options are:
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control planeChoose a specific Kubernetes version for the control plane.
--service-cidr string     Default: "10.96.0.0/12"
The range of IP address used for service VIPsUse alternative range of IP address for service VIPs.
--service-dns-domain string     Default: "cluster.local"
Alternative domain for servicesUse alternative domain for services, e.g. "myorg.internal".
--apiserver-advertise-address string
The IP address the API server is accessible onThe IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
The port the API server is accessible onPort for the API Server to bind to.
--config string
Path to a kubeadm config file. WARNING: Usage of a configuration file is experimentalPath to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control planeChoose a specific Kubernetes version for the control plane.
--pod-network-cidr string
The range of IP addresses used for the Pod networkSpecify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for bootstrap-token
--kubeconfig-dir string     Default: "/etc/kubernetes"
The path where to save the kubeconfig file.
--skip-token-print
Skip printing of the default bootstrap token generated by 'kubeadm init'.
+ + + +### Options inherited from parent commands + + + + + + + + + + + + + + + + +
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
+ + + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs.md similarity index 93% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs.md index 56d01fba8e..538a7a1b33 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs.md @@ -1,11 +1,15 @@ -Generates certificates for a Kubernetes cluster +Certificate generation ### Synopsis This command is not meant to be run on its own. See list of available subcommands. +``` +kubeadm init phase certs [flags] +``` + ### Options diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_all.md new file mode 100644 index 0000000000..910b94afbe --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_all.md @@ -0,0 +1,96 @@ + +Generates all certificates + +### Synopsis + + +Generates all certificates + +``` +kubeadm init phase certs all [flags] +``` + +### Options + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
--apiserver-advertise-address string
The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-cert-extra-sans stringSlice
Optional extra Subject Alternative Names (SANs) to use for the API Server serving certificate. Can be both IP addresses and DNS names.
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for all
--service-cidr string     Default: "10.96.0.0/12"
Use alternative range of IP address for service VIPs.
--service-dns-domain string     Default: "cluster.local"
Use alternative domain for services, e.g. "myorg.internal".
+ + + +### Options inherited from parent commands + + + + + + + + + + + + + + + + +
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
+ + + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_apiserver-etcd-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver-etcd-client.md similarity index 71% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_apiserver-etcd-client.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver-etcd-client.md index 65ef3d793a..01e798e144 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_apiserver-etcd-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver-etcd-client.md @@ -11,7 +11,7 @@ If both files already exist, kubeadm skips the generation step and existing file Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase certs apiserver-etcd-client [flags] +kubeadm init phase certs apiserver-etcd-client [flags] ``` ### Options @@ -27,14 +27,28 @@ kubeadm alpha phase certs apiserver-etcd-client [flags] --cert-dir string     Default: "/etc/kubernetes/pki" - The path where to save the certificates + The path where to save and store the certificates. --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to kubeadm config file. WARNING: Usage of a configuration file is experimental. + + + + --csr-dir string + + + The path to output the CSRs and private keys to + + + + --csr-only + + + Create CSRs instead of generating certificates diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_apiserver-kubelet-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver-kubelet-client.md similarity index 71% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_apiserver-kubelet-client.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver-kubelet-client.md index 37aeb21fb4..b3e30655c4 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_apiserver-kubelet-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver-kubelet-client.md @@ -11,7 +11,7 @@ If both files already exist, kubeadm skips the generation step and existing file Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase certs apiserver-kubelet-client [flags] +kubeadm init phase certs apiserver-kubelet-client [flags] ``` ### Options @@ -27,14 +27,28 @@ kubeadm alpha phase certs apiserver-kubelet-client [flags] --cert-dir string     Default: "/etc/kubernetes/pki" - The path where to save the certificates + The path where to save and store the certificates. --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to kubeadm config file. WARNING: Usage of a configuration file is experimental. + + + + --csr-dir string + + + The path to output the CSRs and private keys to + + + + --csr-only + + + Create CSRs instead of generating certificates diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_apiserver.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver.md similarity index 66% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_apiserver.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver.md index 13f358c12a..a36c16aee3 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_apiserver.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver.md @@ -1,10 +1,10 @@ -Generates the certificate for serving the kubernetes API +Generates the certificate for serving the Kubernetes API ### Synopsis -Generates the certificate for serving the kubernetes API, and saves them into apiserver.cert and apiserver.key files. +Generates the certificate for serving the Kubernetes API, and saves them into apiserver.cert and apiserver.key files. Default SANs are kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, 10.96.0.1, 127.0.0.1 @@ -13,7 +13,7 @@ If both files already exist, kubeadm skips the generation step and existing file Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase certs apiserver [flags] +kubeadm init phase certs apiserver [flags] ``` ### Options @@ -29,28 +29,42 @@ kubeadm alpha phase certs apiserver [flags] --apiserver-advertise-address string - The IP address the API server is accessible on, to use for the API server serving cert + The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface. --apiserver-cert-extra-sans stringSlice - Optional extra altnames to use for the API server serving cert. Can be both IP addresses and DNS names + Optional extra Subject Alternative Names (SANs) to use for the API Server serving certificate. Can be both IP addresses and DNS names. --cert-dir string     Default: "/etc/kubernetes/pki" - The path where to save the certificates + The path where to save and store the certificates. --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to kubeadm config file. WARNING: Usage of a configuration file is experimental. + + + + --csr-dir string + + + The path to output the CSRs and private keys to + + + + --csr-only + + + Create CSRs instead of generating certificates @@ -64,14 +78,14 @@ kubeadm alpha phase certs apiserver [flags] --service-cidr string     Default: "10.96.0.0/12" - Alternative range of IP address for service VIPs, from which derives the internal API server VIP that will be added to the API Server serving cert + Use alternative range of IP address for service VIPs. --service-dns-domain string     Default: "cluster.local" - Alternative domain for services, to use for the API server serving cert + Use alternative domain for services, e.g. "myorg.internal". diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_ca.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_ca.md similarity index 76% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_ca.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_ca.md index 7f4e19a2d3..257da0ea6a 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_ca.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_ca.md @@ -1,17 +1,17 @@ -Generates the self-signed kubernetes CA to provision identities for other kuberenets components +Generates the self-signed Kubernetes CA to provision identities for other Kubernetes components ### Synopsis -Generates the self-signed kubernetes CA to provision identities for other kuberenets components, and saves them into ca.cert and ca.key files. +Generates the self-signed Kubernetes CA to provision identities for other Kubernetes components, and saves them into ca.cert and ca.key files. If both files already exist, kubeadm skips the generation step and existing files will be used. Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase certs ca [flags] +kubeadm init phase certs ca [flags] ``` ### Options @@ -27,14 +27,14 @@ kubeadm alpha phase certs ca [flags] --cert-dir string     Default: "/etc/kubernetes/pki" - The path where to save the certificates + The path where to save and store the certificates. --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to kubeadm config file. WARNING: Usage of a configuration file is experimental. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_etcd-ca.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-ca.md similarity index 88% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_etcd-ca.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-ca.md index 186905c298..210b0d1dd2 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_etcd-ca.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-ca.md @@ -11,7 +11,7 @@ If both files already exist, kubeadm skips the generation step and existing file Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase certs etcd-ca [flags] +kubeadm init phase certs etcd-ca [flags] ``` ### Options @@ -27,14 +27,14 @@ kubeadm alpha phase certs etcd-ca [flags] --cert-dir string     Default: "/etc/kubernetes/pki" - The path where to save the certificates + The path where to save and store the certificates. --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to kubeadm config file. WARNING: Usage of a configuration file is experimental. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_etcd-healthcheck-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-healthcheck-client.md similarity index 71% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_etcd-healthcheck-client.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-healthcheck-client.md index fd5b135b02..65e5c41bed 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_etcd-healthcheck-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-healthcheck-client.md @@ -11,7 +11,7 @@ If both files already exist, kubeadm skips the generation step and existing file Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase certs etcd-healthcheck-client [flags] +kubeadm init phase certs etcd-healthcheck-client [flags] ``` ### Options @@ -27,14 +27,28 @@ kubeadm alpha phase certs etcd-healthcheck-client [flags] --cert-dir string     Default: "/etc/kubernetes/pki" - The path where to save the certificates + The path where to save and store the certificates. --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to kubeadm config file. WARNING: Usage of a configuration file is experimental. + + + + --csr-dir string + + + The path to output the CSRs and private keys to + + + + --csr-only + + + Create CSRs instead of generating certificates diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_etcd-peer.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-peer.md similarity index 72% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_etcd-peer.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-peer.md index d9a73e800a..2e045ae1e1 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_etcd-peer.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-peer.md @@ -13,7 +13,7 @@ If both files already exist, kubeadm skips the generation step and existing file Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase certs etcd-peer [flags] +kubeadm init phase certs etcd-peer [flags] ``` ### Options @@ -29,14 +29,28 @@ kubeadm alpha phase certs etcd-peer [flags] --cert-dir string     Default: "/etc/kubernetes/pki" - The path where to save the certificates + The path where to save and store the certificates. --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to kubeadm config file. WARNING: Usage of a configuration file is experimental. + + + + --csr-dir string + + + The path to output the CSRs and private keys to + + + + --csr-only + + + Create CSRs instead of generating certificates diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_etcd-server.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-server.md similarity index 69% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_etcd-server.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-server.md index 6965eceb66..a960d01e4f 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_etcd-server.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-server.md @@ -6,14 +6,14 @@ Generates the certificate for serving etcd Generates the certificate for serving etcd, and saves them into etcd/server.cert and etcd/server.key files. -Default SANs are localhost, 127.0.0.1, ::1 +Default SANs are localhost, 127.0.0.1, 127.0.0.1, ::1 If both files already exist, kubeadm skips the generation step and existing files will be used. Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase certs etcd-server [flags] +kubeadm init phase certs etcd-server [flags] ``` ### Options @@ -29,14 +29,28 @@ kubeadm alpha phase certs etcd-server [flags] --cert-dir string     Default: "/etc/kubernetes/pki" - The path where to save the certificates + The path where to save and store the certificates. --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to kubeadm config file. WARNING: Usage of a configuration file is experimental. + + + + --csr-dir string + + + The path to output the CSRs and private keys to + + + + --csr-only + + + Create CSRs instead of generating certificates diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_front-proxy-ca.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_front-proxy-ca.md similarity index 88% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_front-proxy-ca.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_front-proxy-ca.md index e6cde90042..bff25d78cd 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_front-proxy-ca.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_front-proxy-ca.md @@ -11,7 +11,7 @@ If both files already exist, kubeadm skips the generation step and existing file Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase certs front-proxy-ca [flags] +kubeadm init phase certs front-proxy-ca [flags] ``` ### Options @@ -27,14 +27,14 @@ kubeadm alpha phase certs front-proxy-ca [flags] --cert-dir string     Default: "/etc/kubernetes/pki" - The path where to save the certificates + The path where to save and store the certificates. --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to kubeadm config file. WARNING: Usage of a configuration file is experimental. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_front-proxy-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_front-proxy-client.md similarity index 70% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_front-proxy-client.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_front-proxy-client.md index bde1a92cda..9a051340a1 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_front-proxy-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_front-proxy-client.md @@ -11,7 +11,7 @@ If both files already exist, kubeadm skips the generation step and existing file Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase certs front-proxy-client [flags] +kubeadm init phase certs front-proxy-client [flags] ``` ### Options @@ -27,14 +27,28 @@ kubeadm alpha phase certs front-proxy-client [flags] --cert-dir string     Default: "/etc/kubernetes/pki" - The path where to save the certificates + The path where to save and store the certificates. --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to kubeadm config file. WARNING: Usage of a configuration file is experimental. + + + + --csr-dir string + + + The path to output the CSRs and private keys to + + + + --csr-only + + + Create CSRs instead of generating certificates diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_sa.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_sa.md similarity index 68% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_sa.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_sa.md index 4c707b4a5c..951b773eac 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_certs_sa.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_sa.md @@ -9,7 +9,7 @@ Generates the private key for signing service account tokens along with its publ Alpha Disclaimer: this command is currently alpha. ``` -kubeadm alpha phase certs sa [flags] +kubeadm init phase certs sa [flags] ``` ### Options @@ -21,20 +21,6 @@ kubeadm alpha phase certs sa [flags] - - --cert-dir string     Default: "/etc/kubernetes/pki" - - - The path where to save the certificates - - - - --config string - - - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) - - -h, --help diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_controlplane.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane.md similarity index 91% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_controlplane.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane.md index 3605a67b17..38cc40ed90 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_controlplane.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane.md @@ -6,6 +6,10 @@ Generates all static Pod manifest files necessary to establish the control plane This command is not meant to be run on its own. See list of available subcommands. +``` +kubeadm init phase control-plane [flags] +``` + ### Options @@ -19,7 +23,7 @@ This command is not meant to be run on its own. See list of available subcommand - + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_controlplane_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_all.md similarity index 73% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_controlplane_all.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_all.md index e940dae06b..322a545318 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_controlplane_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_all.md @@ -1,26 +1,13 @@ -Generates all static Pod manifest files necessary to establish the control plane +Generates all static Pod manifest files ### Synopsis -Generates all static Pod manifest files necessary to establish the control plane. - -Alpha Disclaimer: this command is currently alpha. +Generates all static Pod manifest files ``` -kubeadm alpha phase controlplane all [flags] -``` - -### Examples - -``` - # Generates all static Pod manifest files for control plane components, - # functionally equivalent to what generated by kubeadm init. - kubeadm alpha phase controlplane all - - # Generates all static Pod manifest files using options read from a configuration file. - kubeadm alpha phase controlplane --config masterconfiguration.yaml +kubeadm init phase control-plane all [flags] ``` ### Options @@ -36,14 +23,14 @@ kubeadm alpha phase controlplane all [flags] - + - + @@ -57,14 +44,14 @@ kubeadm alpha phase controlplane all [flags] - + - + @@ -78,7 +65,7 @@ kubeadm alpha phase controlplane all [flags] - + @@ -88,18 +75,25 @@ kubeadm alpha phase controlplane all [flags] + + + + + + + - + - + @@ -113,7 +107,7 @@ kubeadm alpha phase controlplane all [flags] - + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_controlplane_apiserver.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_apiserver.md similarity index 74% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_controlplane_apiserver.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_apiserver.md index de5fff7083..6ebc0d2724 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_controlplane_apiserver.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_apiserver.md @@ -1,15 +1,13 @@ -Generates the API server static Pod manifest +Generates the kube-apiserver static Pod manifest ### Synopsis -Generates the static Pod manifest file for the API server and saves it into /etc/kubernetes/manifests/kube-apiserver.yaml file. - -Alpha Disclaimer: this command is currently alpha. +Generates the kube-apiserver static Pod manifest ``` -kubeadm alpha phase controlplane apiserver [flags] +kubeadm init phase control-plane apiserver [flags] ``` ### Options @@ -25,14 +23,14 @@ kubeadm alpha phase controlplane apiserver [flags] - + - + @@ -46,21 +44,21 @@ kubeadm alpha phase controlplane apiserver [flags] - + - + - + @@ -70,18 +68,25 @@ kubeadm alpha phase controlplane apiserver [flags] + + + + + + + - + - + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_controlplane_controller-manager.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_controller-manager.md similarity index 71% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_controlplane_controller-manager.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_controller-manager.md index 402703698c..19fb762a7e 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_controlplane_controller-manager.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_controller-manager.md @@ -1,15 +1,13 @@ -Generates the controller-manager static Pod manifest +Generates the kube-controller-manager static Pod manifest ### Synopsis -Generates the static Pod manifest file for the controller-manager and saves it into /etc/kubernetes/manifests/kube-controller-manager.yaml file. - -Alpha Disclaimer: this command is currently alpha. +Generates the kube-controller-manager static Pod manifest ``` -kubeadm alpha phase controlplane controller-manager [flags] +kubeadm init phase control-plane controller-manager [flags] ``` ### Options @@ -25,14 +23,14 @@ kubeadm alpha phase controlplane controller-manager [flags] - + - + @@ -49,18 +47,25 @@ kubeadm alpha phase controlplane controller-manager [flags] + + + + + + + - + - + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_controlplane_scheduler.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_scheduler.md similarity index 74% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_controlplane_scheduler.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_scheduler.md index e07e4b46d8..9cf579ab56 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_controlplane_scheduler.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_scheduler.md @@ -1,15 +1,13 @@ -Generates the scheduler static Pod manifest +Generates the kube-scheduler static Pod manifest ### Synopsis -Generates the static Pod manifest file for the scheduler and saves it into /etc/kubernetes/manifests/kube-scheduler.yaml file. - -Alpha Disclaimer: this command is currently alpha. +Generates the kube-scheduler static Pod manifest ``` -kubeadm alpha phase controlplane scheduler [flags] +kubeadm init phase control-plane scheduler [flags] ``` ### Options @@ -25,14 +23,14 @@ kubeadm alpha phase controlplane scheduler [flags] - + - + @@ -42,11 +40,18 @@ kubeadm alpha phase controlplane scheduler [flags] + + + + + + + - + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_etcd.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_etcd.md new file mode 100644 index 0000000000..fa2a91f055 --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_etcd.md @@ -0,0 +1,54 @@ + +Generates static Pod manifest file for local etcd. + +### Synopsis + + +This command is not meant to be run on its own. See list of available subcommands. + +``` +kubeadm init phase etcd [flags] +``` + +### Options + +
-h, --help
help for controlplanehelp for control-plane
--apiserver-advertise-address string
The IP address of the API server is accessible onThe IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
The port the API server is accessible onPort for the API Server to bind to.
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where certificates are storedThe path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimentalPath to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--feature-gates string
A set of key=value pairs that describe feature gates for various features. Options are:
Auditing=true|false (ALPHA - default=false)
CoreDNS=true|false (default=true)
DynamicKubeletConfig=true|false (BETA - default=false)		A set of key=value pairs that describe feature gates for various features. Options are:
help for all
--image-repository string     Default: "k8s.gcr.io"
Choose a container registry to pull control plane images from
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control planeChoose a specific Kubernetes version for the control plane.
--pod-network-cidr string
The range of IP addresses used for the Pod networkSpecify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
--service-cidr string     Default: "10.96.0.0/12"
The range of IP address used for service VIPsUse alternative range of IP address for service VIPs.
--apiserver-advertise-address string
The IP address of the API server is accessible onThe IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
The port the API server is accessible onPort for the API Server to bind to.
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where certificates are storedThe path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimentalPath to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--feature-gates string
A set of key=value pairs that describe feature gates for various features. Options are:
Auditing=true|false (ALPHA - default=false)
CoreDNS=true|false (default=true)
DynamicKubeletConfig=true|false (BETA - default=false)		A set of key=value pairs that describe feature gates for various features. Options are:
help for apiserver
--image-repository string     Default: "k8s.gcr.io"
Choose a container registry to pull control plane images from
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control planeChoose a specific Kubernetes version for the control plane.
--service-cidr string     Default: "10.96.0.0/12"
The range of IP address used for service VIPsUse alternative range of IP address for service VIPs.
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where certificates are storedThe path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimentalPath to kubeadm config file. WARNING: Usage of a configuration file is experimental.
help for controller-manager
--image-repository string     Default: "k8s.gcr.io"
Choose a container registry to pull control plane images from
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control planeChoose a specific Kubernetes version for the control plane.
--pod-network-cidr string
The range of IP addresses used for the Pod networkSpecify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where certificates are storedThe path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimentalPath to kubeadm config file. WARNING: Usage of a configuration file is experimental.
help for scheduler
--image-repository string     Default: "k8s.gcr.io"
Choose a container registry to pull control plane images from
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control planeChoose a specific Kubernetes version for the control plane.
+ + + + + + + + + + + + + + +
-h, --help
help for etcd
+ + + +### Options inherited from parent commands + + + + + + + + + + + + + + + + +
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
+ + + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_etcd_local.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_etcd_local.md similarity index 67% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_etcd_local.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_etcd_local.md index 19802040fc..ced1517a3b 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_etcd_local.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_etcd_local.md @@ -1,26 +1,25 @@ -Generates the static Pod manifest file for a local, single-node etcd instance +Generates the static Pod manifest file for a local, single-node local etcd instance. ### Synopsis -Generates the static Pod manifest file for a local, single-node etcd instance and saves it to /etc/kubernetes/manifests/etcd.yaml file. - -Alpha Disclaimer: this command is currently alpha. +Generates the static Pod manifest file for a local, single-node local etcd instance. ``` -kubeadm alpha phase etcd local [flags] +kubeadm init phase etcd local [flags] ``` ### Examples ``` # Generates the static Pod manifest file for etcd, functionally - # equivalent to what generated by kubeadm init. - kubeadm alpha phase etcd local + # equivalent to what is generated by kubeadm init. + kubeadm init phase etcd local - # Generates the static Pod manifest file for etcd. - kubeadm alpha phase etcd local --config masterconfiguration.yaml + # Generates the static Pod manifest file for etcd using options + # read from a configuration file. + kubeadm init phase etcd local --config config.yaml ``` ### Options @@ -36,14 +35,14 @@ kubeadm alpha phase etcd local [flags] --cert-dir string     Default: "/etc/kubernetes/pki" - The path where certificates are stored + The path where to save and store the certificates. --config string - Path to kubeadm config file. WARNING: Usage of a configuration file is experimental + Path to kubeadm config file. WARNING: Usage of a configuration file is experimental. @@ -53,6 +52,13 @@ kubeadm alpha phase etcd local [flags] help for local + + --image-repository string     Default: "k8s.gcr.io" + + + Choose a container registry to pull control plane images from + + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig.md similarity index 95% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig.md index ba11bde684..8931be52e7 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig.md @@ -6,6 +6,10 @@ Generates all kubeconfig files necessary to establish the control plane and the This command is not meant to be run on its own. See list of available subcommands. +``` +kubeadm init phase kubeconfig [flags] +``` + ### Options diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_admin.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_admin.md similarity index 82% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_admin.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_admin.md index e58fad7829..a10f9aaf65 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_admin.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_admin.md @@ -4,12 +4,10 @@ Generates a kubeconfig file for the admin to use and for kubeadm itself ### Synopsis -Generates the kubeconfig file for the admin and for kubeadm itself, and saves it to admin.conf file. - -Alpha Disclaimer: this command is currently alpha. +Generates the kubeconfig file for the admin and for kubeadm itself, and saves it to admin.conf file. ``` -kubeadm alpha phase kubeconfig admin [flags] +kubeadm init phase kubeconfig admin [flags] ``` ### Options @@ -25,28 +23,28 @@ kubeadm alpha phase kubeconfig admin [flags] - + - + - + - + @@ -60,7 +58,7 @@ kubeadm alpha phase kubeconfig admin [flags] - + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_all.md new file mode 100644 index 0000000000..698541d273 --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_all.md @@ -0,0 +1,96 @@ + +Generates all kubeconfig files + +### Synopsis + + +Generates all kubeconfig files + +``` +kubeadm init phase kubeconfig all [flags] +``` + +### Options + +
--apiserver-advertise-address string
The IP address the API server is accessible onThe IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
The port the API server is accessible onPort for the API Server to bind to.
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where certificates are storedThe path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimentalPath to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--kubeconfig-dir string     Default: "/etc/kubernetes"
The path where to save the kubeconfig fileThe path where to save the kubeconfig file.
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
--apiserver-advertise-address string
The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
Port for the API Server to bind to.
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for all
--kubeconfig-dir string     Default: "/etc/kubernetes"
The path where to save the kubeconfig file.
--node-name string
Specify the node name.
+ + + +### Options inherited from parent commands + + + + + + + + + + + + + + + + +
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
+ + + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_controller-manager.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_controller-manager.md similarity index 81% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_controller-manager.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_controller-manager.md index 24f77d50af..3a27c9a950 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_controller-manager.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_controller-manager.md @@ -4,12 +4,10 @@ Generates a kubeconfig file for the controller manager to use ### Synopsis -Generates the kubeconfig file for the controller manager to use and saves it to /etc/kubernetes/controller-manager.conf file. - -Alpha Disclaimer: this command is currently alpha. +Generates the kubeconfig file for the controller manager to use and saves it to controller-manager.conf file ``` -kubeadm alpha phase kubeconfig controller-manager [flags] +kubeadm init phase kubeconfig controller-manager [flags] ``` ### Options @@ -25,28 +23,28 @@ kubeadm alpha phase kubeconfig controller-manager [flags] --apiserver-advertise-address string - The IP address the API server is accessible on + The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface. --apiserver-bind-port int32     Default: 6443 - The port the API server is accessible on + Port for the API Server to bind to. --cert-dir string     Default: "/etc/kubernetes/pki" - The path where certificates are stored + The path where to save and store the certificates. --config string - Path to kubeadm config file. WARNING: Usage of a configuration file is experimental + Path to kubeadm config file. WARNING: Usage of a configuration file is experimental. @@ -60,7 +58,7 @@ kubeadm alpha phase kubeconfig controller-manager [flags] --kubeconfig-dir string     Default: "/etc/kubernetes" - The path where to save the kubeconfig file + The path where to save the kubeconfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_kubelet.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_kubelet.md similarity index 73% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_kubelet.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_kubelet.md index ff7c787b1d..2ba61026cb 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_kubelet.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_kubelet.md @@ -1,17 +1,15 @@ -Generates a kubeconfig file for the kubelet to use. Please note that this should be used *only* for bootstrapping purposes +Generates a kubeconfig file for the kubelet to use *only* for cluster bootstrapping purposes ### Synopsis -Generates the kubeconfig file for the kubelet to use and saves it to /etc/kubernetes/kubelet.conf file. +Generates the kubeconfig file for the kubelet to use and saves it to kubelet.conf file. -Please note that this should only be used for bootstrapping purposes. After your control plane is up, you should request all kubelet credentials from the CSR API. - -Alpha Disclaimer: this command is currently alpha. +Please note that this should only be used for cluster bootstrapping purposes. After your control plane is up, you should request all kubelet credentials from the CSR API. ``` -kubeadm alpha phase kubeconfig kubelet [flags] +kubeadm init phase kubeconfig kubelet [flags] ``` ### Options @@ -27,28 +25,28 @@ kubeadm alpha phase kubeconfig kubelet [flags] --apiserver-advertise-address string - The IP address the API server is accessible on + The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface. --apiserver-bind-port int32     Default: 6443 - The port the API server is accessible on + Port for the API Server to bind to. --cert-dir string     Default: "/etc/kubernetes/pki" - The path where certificates are stored + The path where to save and store the certificates. --config string - Path to kubeadm config file. WARNING: Usage of a configuration file is experimental + Path to kubeadm config file. WARNING: Usage of a configuration file is experimental. @@ -62,14 +60,14 @@ kubeadm alpha phase kubeconfig kubelet [flags] --kubeconfig-dir string     Default: "/etc/kubernetes" - The path where to save the kubeconfig file + The path where to save the kubeconfig file. --node-name string - The node name that should be used for the kubelet client certificate + Specify the node name. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_scheduler.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_scheduler.md similarity index 82% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_scheduler.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_scheduler.md index 6de62a33ce..892e77c829 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubeconfig_scheduler.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_scheduler.md @@ -4,12 +4,10 @@ Generates a kubeconfig file for the scheduler to use ### Synopsis -Generates the kubeconfig file for the scheduler to use and saves it to /etc/kubernetes/scheduler.conf file. - -Alpha Disclaimer: this command is currently alpha. +Generates the kubeconfig file for the scheduler to use and saves it to scheduler.conf file. ``` -kubeadm alpha phase kubeconfig scheduler [flags] +kubeadm init phase kubeconfig scheduler [flags] ``` ### Options @@ -25,28 +23,28 @@ kubeadm alpha phase kubeconfig scheduler [flags] --apiserver-advertise-address string - The IP address the API server is accessible on + The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface. --apiserver-bind-port int32     Default: 6443 - The port the API server is accessible on + Port for the API Server to bind to. --cert-dir string     Default: "/etc/kubernetes/pki" - The path where certificates are stored + The path where to save and store the certificates. --config string - Path to kubeadm config file. WARNING: Usage of a configuration file is experimental + Path to kubeadm config file. WARNING: Usage of a configuration file is experimental. @@ -60,7 +58,7 @@ kubeadm alpha phase kubeconfig scheduler [flags] --kubeconfig-dir string     Default: "/etc/kubernetes" - The path where to save the kubeconfig file + The path where to save the kubeconfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_mark-master.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubelet-start.md similarity index 58% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_mark-master.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubelet-start.md index 61e3e17473..69a28ad329 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_mark-master.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubelet-start.md @@ -1,25 +1,20 @@ -Mark a node as master +Writes kubelet settings and (re)starts the kubelet ### Synopsis -Applies a label that specifies that a node is a master and a taint that forces workloads to be deployed accordingly. - -Alpha Disclaimer: this command is currently alpha. +Writes a file with KubeletConfiguration and an environment file with node specific kubelet settings, and then (re)starts kubelet. ``` -kubeadm alpha phase mark-master [flags] +kubeadm init phase kubelet-start [flags] ``` ### Examples ``` - # Applies master label and taint to the current node, functionally equivalent to what executed by kubeadm init. - kubeadm alpha phase mark-master - - # Applies master label and taint to a specific node - kubeadm alpha phase mark-master --node-name myNode + # Writes a dynamic environment file with kubelet flags from a InitConfiguration file. + kubeadm init phase kubelet-start --config masterconfig.yaml ``` ### Options @@ -35,28 +30,28 @@ kubeadm alpha phase mark-master [flags] --config string - Path to kubeadm config file. WARNING: Usage of a configuration file is experimental + Path to kubeadm config file. WARNING: Usage of a configuration file is experimental. + + + + --cri-socket string     Default: "/var/run/dockershim.sock" + + + Specify the CRI socket to connect to. -h, --help - help for mark-master - - - - --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - - - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + help for kubelet-start --node-name string - The node name to which label and taints should apply + Specify the node name. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_mark-control-plane.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_mark-control-plane.md new file mode 100644 index 0000000000..4c0d119be5 --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_mark-control-plane.md @@ -0,0 +1,78 @@ + +Mark a node as a control-plane + +### Synopsis + + +Mark a node as a control-plane + +``` +kubeadm init phase mark-control-plane [flags] +``` + +### Examples + +``` + # Applies control-plane label and taint to the current node, functionally equivalent to what executed by kubeadm init. + kubeadm init phase mark-control-plane --config config.yml + + # Applies control-plane label and taint to a specific node + kubeadm init phase mark-control-plane --node-name myNode +``` + +### Options + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for mark-control-plane
--node-name string
Specify the node name.
+ + + +### Options inherited from parent commands + + + + + + + + + + + + + + + + +
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
+ + + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_preflight_master.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_preflight.md similarity index 81% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_preflight_master.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_preflight.md index eb1b8de7ef..b5f2626e72 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_preflight_master.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_preflight.md @@ -4,19 +4,17 @@ Run master pre-flight checks ### Synopsis -Run master pre-flight checks, functionally equivalent to what implemented by kubeadm init. - -Alpha Disclaimer: this command is currently alpha. +Run master pre-flight checks, functionally equivalent to what implemented by kubeadm init. ``` -kubeadm alpha phase preflight master [flags] +kubeadm init phase preflight [flags] ``` ### Examples ``` - # Run master pre-flight checks. - kubeadm alpha phase preflight master + # Run master pre-flight checks using a config file. + kubeadm init phase preflight --config kubeadm-config.yml ``` ### Options @@ -28,11 +26,25 @@ kubeadm alpha phase preflight master [flags] + + --config string + + + Path to kubeadm config file. WARNING: Usage of a configuration file is experimental. + + -h, --help - help for master + help for preflight + + + + --ignore-preflight-errors stringSlice + + + A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks. @@ -49,20 +61,6 @@ kubeadm alpha phase preflight master [flags] - - --config string - - - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) - - - - --ignore-preflight-errors stringSlice - - - A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks. - - --rootfs string diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config.md similarity index 85% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config.md index 11771b9355..0f20e04b16 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config.md @@ -1,11 +1,15 @@ -Invoke subsets of kubeadm functions separately for a manual install. +Uploads the kubeadm and kubelet configuration to a ConfigMap ### Synopsis This command is not meant to be run on its own. See list of available subcommands. +``` +kubeadm init phase upload-config [flags] +``` + ### Options @@ -19,7 +23,7 @@ This command is not meant to be run on its own. See list of available subcommand - + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_all.md similarity index 70% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_all.md index 638eda6f50..8e697d5c5b 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_bootstrap-token.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_all.md @@ -1,10 +1,14 @@ -Manage kubeadm-specific bootstrap token functions +Uploads all configuration to a config map ### Synopsis -This command is not meant to be run on its own. See list of available subcommands. +Uploads all configuration to a config map + +``` +kubeadm init phase upload-config all [flags] +``` ### Options @@ -15,18 +19,25 @@ This command is not meant to be run on its own. See list of available subcommand + + + + + + + - + - + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config_annotate-cri.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubeadm.md similarity index 66% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config_annotate-cri.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubeadm.md index 5b9cd0fa6a..986d5ac0ad 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config_annotate-cri.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubeadm.md @@ -1,21 +1,22 @@ -annotates the node with the given crisocket +Uploads the kubeadm ClusterConfiguration to a ConfigMap ### Synopsis -Adds an annotation to the current node with the CRI socket specified in the kubeadm InitConfiguration object. +Uploads the kubeadm ClusterConfiguration to a ConfigMap called kubeadm-config in the kube-system namespace. This enables correct configuration of system components and a seamless user experience when upgrading. -Alpha Disclaimer: this command is currently alpha. +Alternatively, you can use kubeadm config. ``` -kubeadm alpha phase kubelet config annotate-cri [flags] +kubeadm init phase upload-config kubeadm [flags] ``` ### Examples ``` - kubeadm alpha phase kubelet config annotate-cri --config kubeadm.yaml + # uploads the configuration of your cluster + kubeadm init phase upload-config --config=myConfig.yaml ``` ### Options @@ -31,21 +32,21 @@ kubeadm alpha phase kubelet config annotate-cri [flags] - + - + - + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config_upload.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubelet.md similarity index 76% rename from content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config_upload.md rename to content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubelet.md index 63584465ca..8eb48c01ec 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_phase_kubelet_config_upload.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubelet.md @@ -1,22 +1,20 @@ -Uploads kubelet configuration to a ConfigMap based on a kubeadm InitConfiguration file. +Uploads the kubelet component config to a ConfigMap ### Synopsis -Uploads kubelet configuration extracted from the kubeadm InitConfiguration object to a ConfigMap of the form kubelet-config-1.X in the cluster, where X is the minor version of the current (API Server) Kubernetes version. - -Alpha Disclaimer: this command is currently alpha. +Uploads kubelet configuration extracted from the kubeadm InitConfiguration object to a ConfigMap of the form kubelet-config-1.X in the cluster, where X is the minor version of the current (API Server) Kubernetes version. ``` -kubeadm alpha phase kubelet config upload [flags] +kubeadm init phase upload-config kubelet [flags] ``` ### Examples ``` # Uploads the kubelet configuration from the kubeadm Config file to a ConfigMap in the cluster. - kubeadm alpha phase kubelet config upload --config kubeadm.yaml + kubeadm init phase upload-config kubelet --config kubeadm.yaml ``` ### Options @@ -32,21 +30,21 @@ kubeadm alpha phase kubelet config upload [flags] - + - + - + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join.md index cc288daa97..5116ad9850 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join.md @@ -16,7 +16,7 @@ provide a file - a subset of the standard kubeconfig file. This file can be a local file or downloaded via an HTTPS URL. The forms are kubeadm join --discovery-token abcdef.1234567890abcdef 1.2.3.4:6443, kubeadm join --discovery-file path/to/file.conf, or kubeadm join ---discovery-file `https://url/file.conf`. Only one form can be used. If +--discovery-file https://url/file.conf. Only one form can be used. If the discovery information is loaded from a URL, HTTPS must be used. Also, in that case the host installed CA bundle is used to verify the connection. @@ -92,14 +92,14 @@ kubeadm join [flags] - + - + @@ -123,13 +123,6 @@ kubeadm join [flags] - - - - - - - @@ -151,18 +144,11 @@ kubeadm join [flags] - - - - - - - - + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset.md index d4f273f1c8..ee31960853 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset.md @@ -54,6 +54,13 @@ kubeadm reset [flags] + + + + + + +
-h, --help
help for phasehelp for upload-config
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for bootstrap-tokenhelp for all
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--config string
Path to kubeadm config file (WARNING: Usage of a configuration file is experimental)Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for annotate-crihelp for kubeadm
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--config string
Path to kubeadm config file (WARNING: Usage of a configuration file is experimental)Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for uploadhelp for kubelet
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--discovery-file string
A file or url from which to load cluster information.A file or URL from which to load cluster information.
--discovery-token string
A token used to validate cluster information fetched from the api server.A token used to validate cluster information fetched from the API server.
Create a new control plane instance on this node
--feature-gates string
A set of key=value pairs that describe feature gates for various features. Options are:
Auditing=true|false (ALPHA - default=false)
CoreDNS=true|false (default=true)
DynamicKubeletConfig=true|false (BETA - default=false)
-h, --help
Specify the node name.
--tls-bootstrap-token string
A token used for TLS bootstrapping.
--token string
Use this token for both discovery-token and tls-bootstrap-token.Use this token for both discovery-token and tls-bootstrap-token when those values are not provided.
A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token.md index 77e2596f6b..3ba6f21456 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token.md @@ -54,7 +54,7 @@ kubeadm token [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_create.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_create.md index fe2a710103..dfb557fcc6 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_create.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_create.md @@ -100,7 +100,7 @@ kubeadm token create [token] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_delete.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_delete.md index 3a92d9ff74..415ce6e107 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_delete.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_delete.md @@ -56,7 +56,7 @@ kubeadm token delete [token-value] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_generate.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_generate.md index 43aed13f1c..2194e64203 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_generate.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_generate.md @@ -61,7 +61,7 @@ kubeadm token generate [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_list.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_list.md index ed4f822fd4..d49fb72c86 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_list.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_list.md @@ -53,7 +53,7 @@ kubeadm token list [flags] --kubeconfig string     Default: "/etc/kubernetes/admin.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_apply.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_apply.md index 87ace5dcc0..b199ac406f 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_apply.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_apply.md @@ -37,7 +37,7 @@ kubeadm upgrade apply [version] --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to a kubeadm configuration file. @@ -65,7 +65,7 @@ kubeadm upgrade apply [version] --feature-gates string - A set of key=value pairs that describe feature gates for various features. Options are:
Auditing=true|false (ALPHA - default=false)
CoreDNS=true|false (default=true)
DynamicKubeletConfig=true|false (BETA - default=false) + A set of key=value pairs that describe feature gates for various features. Options are:
@@ -97,10 +97,10 @@ kubeadm upgrade apply [version] - --kubeconfig string     Default: "/Users/zarnold/.kube/config" + --kubeconfig string     Default: "/Users/tim/.kube/config" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_diff.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_diff.md index 099dd2381c..0403034bdf 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_diff.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_diff.md @@ -30,7 +30,7 @@ kubeadm upgrade diff [version] [flags] --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to a kubeadm configuration file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_config.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_config.md index 957e151227..85c5ccd8fe 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_config.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_config.md @@ -14,11 +14,11 @@ kubeadm upgrade node config [flags] ``` # Downloads the kubelet configuration from the ConfigMap in the cluster. Uses a specific desired kubelet version. - kubeadm upgrade node config --kubelet-version v1.12.0 + kubeadm upgrade node config --kubelet-version v1.13.0 # Simulates the downloading of the kubelet configuration from the ConfigMap in the cluster with a specific desired # version. Does not change any state locally on the node. - kubeadm upgrade node config --kubelet-version v1.12.0 --dry-run + kubeadm upgrade node config --kubelet-version v1.13.0 --dry-run ``` ### Options @@ -48,7 +48,7 @@ kubeadm upgrade node config [flags] --kubeconfig string     Default: "/etc/kubernetes/kubelet.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_experimental-control-plane.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_experimental-control-plane.md index 091f7ac1ed..7b85992770 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_experimental-control-plane.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_experimental-control-plane.md @@ -14,11 +14,11 @@ kubeadm upgrade node experimental-control-plane [flags] ``` # Downloads the kubelet configuration from the ConfigMap in the cluster. Uses a specific desired kubelet version. - kubeadm upgrade node config --kubelet-version v1.12.0 + kubeadm upgrade node config --kubelet-version v1.13.0 # Simulates the downloading of the kubelet configuration from the ConfigMap in the cluster with a specific desired # version. Does not change any state locally on the node. - kubeadm upgrade node config --kubelet-version v1.12.0 --dry-run + kubeadm upgrade node config --kubelet-version v1.13.0 --dry-run ``` ### Options @@ -37,6 +37,13 @@ kubeadm upgrade node experimental-control-plane [flags] Do not change any state, just output the actions that would be performed. + + --etcd-upgrade     Default: true + + + Perform the upgrade of etcd. + + -h, --help @@ -48,7 +55,7 @@ kubeadm upgrade node experimental-control-plane [flags] --kubeconfig string     Default: "/etc/kubernetes/kubelet.conf" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_plan.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_plan.md index 842e070f19..a58b22d6ba 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_plan.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_plan.md @@ -37,14 +37,14 @@ kubeadm upgrade plan [version] [flags] --config string - Path to kubeadm config file (WARNING: Usage of a configuration file is experimental) + Path to a kubeadm configuration file. --feature-gates string - A set of key=value pairs that describe feature gates for various features. Options are:
Auditing=true|false (ALPHA - default=false)
CoreDNS=true|false (default=true)
DynamicKubeletConfig=true|false (BETA - default=false) + A set of key=value pairs that describe feature gates for various features. Options are:
@@ -62,10 +62,10 @@ kubeadm upgrade plan [version] [flags] - --kubeconfig string     Default: "/Users/zarnold/.kube/config" + --kubeconfig string     Default: "/Users/tim/.kube/config" - The KubeConfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. + The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file. diff --git a/content/en/docs/reference/setup-tools/kubeadm/implementation-details.md b/content/en/docs/reference/setup-tools/kubeadm/implementation-details.md index 36cb28b8e2..b5c6ebd32c 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/implementation-details.md +++ b/content/en/docs/reference/setup-tools/kubeadm/implementation-details.md @@ -71,7 +71,7 @@ in a majority of cases, and the most intuitive location; other constants paths a The `kubeadm init` [internal workflow](/docs/reference/setup-tools/kubeadm/kubeadm-init/#init-workflow) consists of a sequence of atomic work tasks to perform, as described in `kubeadm init`. -The [`kubeadm alpha phase`](/docs/reference/setup-tools/kubeadm/kubeadm-alpha/) command allows users to invoke individually each task, and ultimately offers a reusable and composable +The [`kubeadm init phase`](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/) command allows users to invoke individually each task, and ultimately offers a reusable and composable API/toolbox that can be used by other Kubernetes bootstrap tools, by any IT automation tool or by advanced user for creating custom clusters. @@ -122,7 +122,7 @@ In any case the user can skip specific preflight checks (or eventually all prefl Please note that: -1. Preflight checks can be invoked individually with the [`kubeadm alpha phase preflight`](/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-preflight) command +1. Preflight checks can be invoked individually with the [`kubeadm init phase preflight`](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-preflight) command ### Generate the necessary certificates @@ -158,7 +158,7 @@ Certificates are stored by default in `/etc/kubernetes/pki`, but this directory 3. If kubeadm is running in [ExternalCA mode](/docs/reference/setup-tools/kubeadm/kubeadm-init/#external-ca-mode); all the certificates must be provided by the user, because kubeadm cannot generate them by itself 4. In case of kubeadm is executed in the `--dry-run` mode, certificates files are written in a temporary folder -5. Certificate generation can be invoked individually with the [`kubeadm alpha phase certs all`](/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-certs) command +5. Certificate generation can be invoked individually with the [`kubeadm init phase certs all`](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-certs) command ### Generate kubeconfig files for control plane components @@ -186,7 +186,7 @@ Please note that: 2. If a given kubeconfig file exists, and its content is evaluated compliant with the above specs, the existing file will be used and the generation phase for the given kubeconfig skipped 3. If kubeadm is running in [ExternalCA mode](/docs/reference/setup-tools/kubeadm/kubeadm-init/#external-ca-mode), all the required kubeconfig must be provided by the user as well, because kubeadm cannot generate any of them by itself 4. In case of kubeadm is executed in the `--dry-run` mode, kubeconfig files are written in a temporary folder -5. Kubeconfig files generation can be invoked individually with the [`kubeadm alpha phase kubeconfig all`](/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-kubeconfig) command +5. Kubeconfig files generation can be invoked individually with the [`kubeadm init phase kubeconfig all`](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-kubeconfig) command ### Generate static Pod manifests for control plane components @@ -213,7 +213,7 @@ Please note that: should be used for all control plane components, this one will be used. see [using custom images](/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-images) for more details 2. In case of kubeadm is executed in the `--dry-run` mode, static Pods files are written in a temporary folder -3. Static Pod manifest generation for master components can be invoked individually with the [`kubeadm alpha phase controlplane all`](/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-controlplane) command +3. Static Pod manifest generation for master components can be invoked individually with the [`kubeadm init phase control-plane all`](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-control-plane) command #### API server @@ -226,10 +226,6 @@ The static Pod manifest for the API server is affected by following parameters p if an external etcd server is not be provided, a local etcd will be used (via host network) - If a cloud provider is specified, the corresponding `--cloud-provider` is configured, together with the `--cloud-config` path if such file exists (this is experimental, alpha and will be removed in a future version) - - If kubeadm is invoked with `--feature-gates=HighAvailability`, the flag `--endpoint-reconciler-type=lease` is set, thus enabling - automatic reconciliation of endpoints for the internal API server VIP - - If kubeadm is invoked with `--feature-gates=DynamicKubeletConfig`, the corresponding feature on API server is activated - with the `--feature-gates=DynamicKubeletConfig=true` flag Other API server flags that are set unconditionally are: @@ -309,11 +305,11 @@ Please note that: 1. The etcd image will be pulled from `k8s.gcr.io`. In case an alternative image repository is specified this one will be used; In case an alternative image name is specified, this one will be used. see [using custom images](/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-images) for more details 2. in case of kubeadm is executed in the `--dry-run` mode, the etcd static Pod manifest is written in a temporary folder -3. Static Pod manifest generation for local etcd can be invoked individually with the [`kubeadm alpha phase etcd local`](/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-etcd) command +3. Static Pod manifest generation for local etcd can be invoked individually with the [`kubeadm init phase etcd local`](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-etcd) command -### (optional and alpha in v1.9) Write init kubelet configuration +### Optional Dynamic Kublet Configuration -If kubeadm is invoked with `--feature-gates=DynamicKubeletConfig`, it writes the kubelet init configuration +To use this functionality call `kubeadm alpha kubelet config enable-dynamic`. It writes the kubelet init configuration into `/var/lib/kubelet/config/init/kubelet` file. The init configuration is used for starting the kubelet on this specific node, providing an alternative for the kubelet drop-in file; @@ -345,7 +341,7 @@ If kubeadm is invoked with `--feature-gates=DynamicKubeletConfig`: 1. Write the kubelet base configuration into the `kubelet-base-config-v1.9` ConfigMap in the `kube-system` namespace 2. Creates RBAC rules for granting read access to that ConfigMap to all bootstrap tokens and all kubelet instances (that is `system:bootstrappers:kubeadm:default-node-token` and `system:nodes` groups) -3. Enable the dynamic kubelet configuration feature for the initial master node by pointing `Node.spec.configSource` to the newly-created ConfigMap +3. Enable the dynamic kubelet configuration feature for the initial control-plane node by pointing `Node.spec.configSource` to the newly-created ConfigMap ### Save the kubeadm ClusterConfiguration in a ConfigMap for later reference @@ -358,7 +354,7 @@ state and make new decisions based on that data. Please note that: 1. Before uploading, sensitive information like e.g. the token are stripped from the configuration -2. Upload of master configuration can be invoked individually with the [`kubeadm alpha phase upload-config`](/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-upload-config) command +2. Upload of master configuration can be invoked individually with the [`kubeadm init phase upload-config`](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-upload-config) command 3. If you initialized your cluster using kubeadm v1.7.x or lower, you must create manually the master configuration ConfigMap before `kubeadm upgrade` to v1.8 . In order to facilitate this task, the [`kubeadm config upload (from-flags|from-file)`](/docs/reference/setup-tools/kubeadm/kubeadm-config/) was implemented @@ -372,7 +368,7 @@ As soon as the control plane is available, kubeadm executes following actions: Please note that: -1. Mark master phase can be invoked individually with the [`kubeadm alpha phase mark-master`](/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-mark-master) command +1. Mark control-plane phase phase can be invoked individually with the [`kubeadm init phase mark-control-plane`](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-mark-master) command ### Configure TLS-Bootstrapping for node joining @@ -383,7 +379,7 @@ existing cluster; for more details see also [design proposal](https://github.com setting API server and controller flags as already described in previous paragraphs. Please note that: -1. TLS bootstrapping for nodes can be configured with the [`kubeadm alpha phase bootstrap-token all`](/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-bootstrap-token) +1. TLS bootstrapping for nodes can be configured with the [`kubeadm init phase bootstrap-token`](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-bootstrap-token) command, executing all the configuration steps described in following paragraphs; alternatively, each step can be invoked individually #### Create a bootstrap token @@ -441,7 +437,7 @@ can handle to serving the `cluster-info` ConfigMap. Kubeadm installs the internal DNS server and the kube-proxy addon components via the API server. Please note that: -1. This phase can be invoked individually with the [`kubeadm alpha phase addon all`](/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-addon) command. +1. This phase can be invoked individually with the [`kubeadm init phase addon all`](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-addon) command. #### proxy @@ -457,9 +453,11 @@ Note that: - The CoreDNS service is named `kube-dns`. This is done to prevent any interruption in service when the user is switching the cluster DNS from kube-dns to CoreDNS or vice-versa -- In Kubernetes version 1.11 and later, CoreDNS is the default DNS server and you must -invoke kubeadm with `--feature-gates=CoreDNS=false` to install kube-dns instead - In Kubernetes version 1.10 and earlier, you must enable CoreDNS with `--feature-gates=CoreDNS=true` +- In Kubernetes version 1.11 and 1.12, CoreDNS is the default DNS server and you must +invoke kubeadm with `--feature-gates=CoreDNS=false` to install kube-dns instead +- In Kubernetes version 1.13 and later, the `CoreDNS` feature gate is no longer available and kube-dns can be installed using the `--config` method described [here](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-addon) + A ServiceAccount for CoreDNS/kube-dns is created in the `kube-system` namespace. @@ -468,11 +466,11 @@ Deploy the `kube-dns` Deployment and Service: - It's the upstream CoreDNS deployment relatively unmodified - The `kube-dns` ServiceAccount is bound to the privileges in the `system:kube-dns` ClusterRole -### (Optional and alpha in v1.9) self-hosting +### Optional self-hosting -This phase is performed only if `kubeadm init` is invoked with `—features-gates=selfHosting` +To enable self hosting on a existing static Pod control-plane use `kubeadm alpha selfhosting pivot`. -The self hosting phase basically replaces static Pods for control plane components with DaemonSets; this is achieved by executing +Self hosting basically replaces static Pods for control plane components with DaemonSets; this is achieved by executing following procedure for API server, scheduler and controller manager static Pods: - Load the static Pod specification from disk @@ -485,21 +483,9 @@ following procedure for API server, scheduler and controller manager static Pods - Create the DaemonSet resource in `kube-system` namespace. Wait until the Pods are running. - Remove the static Pod manifest file. The kubelet will stop the original static Pod-hosted component that was running -Please note that: - -1. Self hosting is not yet resilient to node restarts; this can be fixed with external checkpointing or with kubelet checkpointing +Please note that self hosting is not yet resilient to node restarts; this can be fixed with external checkpointing or with kubelet checkpointing for the control plane Pods. See [self-hosting](/docs/reference/setup-tools/kubeadm/kubeadm-init/#self-hosting) for more details. -2. If invoked with `—features-gates=StoreCertsInSecrets` following additional steps will be executed - - - Creation of `ca`, `apiserver`, `apiserver-kubelet-client`, `sa`, `front-proxy-ca`, `front-proxy-client` TLS secrets - in `kube-system` namespace with respective certificates and keys. - Important! storing the CA key in a Secret might have security implications - - Creation of `schedler.conf` and `controller-manager.conf` secrets in`kube-system` namespace with respective kubeconfig files - - Mutation of all the Pod specs by replacing host path volumes with projected volumes from the secrets above - -3. This phase can be invoked individually with the [`kubeadm alpha phase selfhosting convert-from-staticpods`](/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-self-hosting) command. - ## kubeadm join phases internal design Similarly to `kubeadm init`, also `kubeadm join` internal workflow consists of a sequence of atomic work tasks to perform. diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md index 68c14d99d4..da92919353 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md @@ -11,162 +11,64 @@ weight: 90 from the community. Please try it out and give us feedback! {{< /caution >}} -In v1.8.0, kubeadm introduced the `kubeadm alpha phase` command with the aim of making kubeadm more modular. This modularity enables you to invoke atomic sub-steps of the bootstrap process; you can let kubeadm do some parts and fill in yourself where you need customizations. - -`kubeadm alpha phase` is consistent with [kubeadm init workflow](/docs/reference/setup-tools/kubeadm/kubeadm-init/#init-workflow), -and behind the scene both use the same code. - -## kubeadm alpha phase preflight {#cmd-phase-preflight} - -You can execute preflight checks both for the master node, like in `kubeadm init`, or for the worker node -like in `kubeadm join`. - -{{< tabs name="tab-preflight" >}} -{{< tab name="master" include="generated/kubeadm_alpha_phase_preflight_master.md" />}} -{{< tab name="node" include="generated/kubeadm_alpha_phase_preflight_node.md" />}} -{{< /tabs >}} - - -## kubeadm alpha phase certs {#cmd-phase-certs} - -You can create all required certificates with the `all` subcommand or selectively create certificates. - -{{< tabs name="tab-certs" >}} -{{< tab name="all" include="generated/kubeadm_alpha_phase_certs_all.md" />}} -{{< tab name="ca" include="generated/kubeadm_alpha_phase_certs_ca.md" />}} -{{< tab name="apiserver" include="generated/kubeadm_alpha_phase_certs_apiserver.md" />}} -{{< tab name="apiserver-kubelet-client" include="generated/kubeadm_alpha_phase_certs_apiserver-kubelet-client.md" />}} -{{< tab name="sa" include="generated/kubeadm_alpha_phase_certs_sa.md" />}} -{{< tab name="front-proxy-ca" include="generated/kubeadm_alpha_phase_certs_front-proxy-ca.md" />}} -{{< tab name="front-proxy-client" include="generated/kubeadm_alpha_phase_certs_front-proxy-client.md" />}} -{{< /tabs >}} - -## kubeadm alpha phase certs renew {#cmd-phase-certs-renew} +## kubeadm alpha certs renew {#cmd-certs-renew} You can renew all Kubernetes certificates using the `all` subcommand or renew them selectively. {{< tabs name="tab-certs-renew" >}} -{{< tab name="all" include="generated/kubeadm_alpha_phase_certs_renew_all.md" />}} -{{< tab name="apiserver-etcd-client" include="generated/kubeadm_alpha_phase_certs_renew_apiserver-etcd-client.md" />}} -{{< tab name="apiserver-kubelet-client" include="generated/kubeadm_alpha_phase_certs_renew_apiserver-kubelet-client.md" />}} -{{< tab name="apiserver" include="generated/kubeadm_alpha_phase_certs_renew_apiserver.md" />}} -{{< tab name="etcd-healthcheck-client" include="generated/kubeadm_alpha_phase_certs_renew_etcd-healthcheck-client.md" />}} -{{< tab name="etcd-peer" include="generated/kubeadm_alpha_phase_certs_renew_etcd-peer.md" />}} -{{< tab name="etcd-server" include="generated/kubeadm_alpha_phase_certs_renew_etcd-server.md" />}} -{{< tab name="front-proxy-client" include="generated/kubeadm_alpha_phase_certs_renew_front-proxy-client.md" />}} +{{< tab name="renew" include="generated/kubeadm_alpha_certs_renew.md" />}} +{{< tab name="all" include="generated/kubeadm_alpha_certs_renew_all.md" />}} +{{< tab name="apiserver-etcd-client" include="generated/kubeadm_alpha_certs_renew_apiserver-etcd-client.md" />}} +{{< tab name="apiserver-kubelet-client" include="generated/kubeadm_alpha_certs_renew_apiserver-kubelet-client.md" />}} +{{< tab name="apiserver" include="generated/kubeadm_alpha_certs_renew_apiserver.md" />}} +{{< tab name="etcd-healthcheck-client" include="generated/kubeadm_alpha_certs_renew_etcd-healthcheck-client.md" />}} +{{< tab name="etcd-peer" include="generated/kubeadm_alpha_certs_renew_etcd-peer.md" />}} +{{< tab name="etcd-server" include="generated/kubeadm_alpha_certs_renew_etcd-server.md" />}} +{{< tab name="front-proxy-client" include="generated/kubeadm_alpha_certs_renew_front-proxy-client.md" />}} {{< /tabs >}} -## kubeadm alpha phase kubeconfig {#cmd-phase-kubeconfig} -You can create all required kubeconfig files with the `all` subcommand, or selectively create the files. -Additionally, the `user` subcommand supports the creation of kubeconfig files for additional users. +## kubeadm alpha kubeconfig user {#cmd-phase-kubeconfig} + +The `user` subcommand can be used for the creation of kubeconfig files for additional users. {{< tabs name="tab-kubeconfig" >}} -{{< tab name="all" include="generated/kubeadm_alpha_phase_kubeconfig_all.md" />}} -{{< tab name="admin" include="generated/kubeadm_alpha_phase_kubeconfig_admin.md" />}} -{{< tab name="kubelet" include="generated/kubeadm_alpha_phase_kubeconfig_kubelet.md" />}} -{{< tab name="controller-manager" include="generated/kubeadm_alpha_phase_kubeconfig_controller-manager.md" />}} -{{< tab name="scheduler" include="generated/kubeadm_alpha_phase_kubeconfig_scheduler.md" />}} -{{< tab name="user" include="generated/kubeadm_alpha_phase_kubeconfig_user.md" />}} +{{< tab name="kubeconfig" include="generated/kubeadm_alpha_kubeconfig.md" />}} +{{< tab name="user" include="generated/kubeadm_alpha_kubeconfig_user.md" />}} {{< /tabs >}} -## kubeadm alpha phase kubelet {#cmd-phase-kubelet} +## kubeadm alpha kubelet config {#cmd-phase-kubelet} -Use the following commands to manage the kubelet phase. +Use the following commands to either download the kubelet configuration from the cluster or +to enable the DynamicKubeletConfiguration feature. {{< tabs name="tab-kubelet" >}} -{{< tab name="config annotate-cri" include="generated/kubeadm_alpha_phase_kubelet_config_annotate-cri.md" />}} -{{< tab name="config download" include="generated/kubeadm_alpha_phase_kubelet_config_download.md" />}} -{{< tab name="config enable-dynamic" include="generated/kubeadm_alpha_phase_kubelet_config_enable-dynamic.md" />}} -{{< tab name="config upload" include="generated/kubeadm_alpha_phase_kubelet_config_upload.md" />}} -{{< tab name="config write-to-disk" include="generated/kubeadm_alpha_phase_kubelet_config_write-to-disk.md" />}} -{{< tab name="write-env-file" include="generated/kubeadm_alpha_phase_kubelet_write-env-file.md" />}} +{{< tab name="kubelet" include="generated/kubeadm_alpha_kubelet.md" />}} +{{< tab name="download" include="generated/kubeadm_alpha_kubelet_config_download.md" />}} +{{< tab name="enable-dynamic" include="generated/kubeadm_alpha_kubelet_config_download.md" />}} {{< /tabs >}} -## kubeadm alpha phase controlplane {#cmd-phase-controlplane} +## kubeadm alpha preflight node {#cmd-phase-preflight} -You can create all required static Pod files for the control plane components with the `all` subcommand, -or selectively create the files. +You can use the `node` sub command to run preflight checks on a worker node. -{{< tabs name="tab-controlplane" >}} -{{< tab name="all" include="generated/kubeadm_alpha_phase_controlplane_all.md" />}} -{{< tab name="apiserver" include="generated/kubeadm_alpha_phase_controlplane_apiserver.md" />}} -{{< tab name="controller-manager" include="generated/kubeadm_alpha_phase_controlplane_controller-manager.md" />}} -{{< tab name="scheduler" include="generated/kubeadm_alpha_phase_controlplane_scheduler.md" />}} +{{< tabs name="tab-preflight" >}} +{{< tab name="preflight" include="generated/kubeadm_alpha_preflight.md" />}} +{{< tab name="node" include="generated/kubeadm_alpha_preflight_node.md" />}} {{< /tabs >}} -## kubeadm alpha phase etcd {#cmd-phase-etcd} +## kubeadm alpha selfhosting pivot {#cmd-selfhosting} -Use the following command to create a self-hosted, local etcd instance based on a static Pod file. +The subcommand `pivot` can be used to conver a static Pod-hosted control plane into a self-hosted one. -{{< tabs name="tab-etcd" >}} -{{< tab name="etcd local" include="generated/kubeadm_alpha_phase_etcd_local.md" />}} -{{< /tabs >}} - - -## kubeadm alpha phase mark-master {#cmd-phase-mark-master} - -Use the following command to label and taint the node with the `node-role.kubernetes.io/master=""` key-value pair. - -{{< tabs name="tab-mark-master" >}} -{{< tab name="mark-master" include="generated/kubeadm_alpha_phase_mark-master.md" />}} -{{< /tabs >}} - - -## kubeadm alpha phase bootstrap-token {#cmd-phase-bootstrap-token} - -Use the following actions to fully configure bootstrap tokens. -You can fully configure bootstrap tokens with the `all` subcommand, -or selectively configure single elements. - -{{< tabs name="tab-bootstrap-token" >}} -{{< tab name="all" include="generated/kubeadm_alpha_phase_bootstrap-token_all.md" />}} -{{< tab name="create" include="generated/kubeadm_alpha_phase_bootstrap-token_create.md" />}} -{{< tab name="cluster-info" include="generated/kubeadm_alpha_phase_bootstrap-token_cluster-info.md " />}} -{{< tab name="node allow-auto-approve" include="generated/kubeadm_alpha_phase_bootstrap-token_node_allow-auto-approve.md" />}} -{{< tab name="node allow-post-csrs" include="generated/kubeadm_alpha_phase_bootstrap-token_node_allow-post-csrs.md" />}} -{{< /tabs >}} - - -## kubeadm alpha phase upload-config {#cmd-phase-upload-config} - -You can use this command to upload the kubeadm configuration to your cluster. -Alternatively, you can use [kubeadm config](/docs/reference/setup-tools/kubeadm/kubeadm-config/). - -{{< tabs name="upload-config" >}} -{{< tab name="mark-master" include="generated/kubeadm_alpha_phase_upload-config.md" />}} -{{< /tabs >}} - - -## kubeadm alpha phase addon {#cmd-phase-addon} - -You can install all the available addons with the `all` subcommand, or -install them selectively. - -{{< note >}} -If `kubeadm` is invoked with `--feature-gates=CoreDNS=false`, kube-dns is installed. -{{< /note >}} - -{{< tabs name="tab-addon" >}} -{{< tab name="all" include="generated/kubeadm_alpha_phase_addon_all.md" />}} -{{< tab name="kube-proxy" include="generated/kubeadm_alpha_phase_addon_kube-proxy.md" />}} -{{< tab name="coredns" include="generated/kubeadm_alpha_phase_addon_coredns.md" />}} -{{< /tabs >}} - - -## kubeadm alpha phase self-hosting {#cmd-phase-self-hosting} - -{{< caution >}} -Self-hosting is an alpha feature. See [kubeadm init](/docs/reference/setup-tools/kubeadm/kubeadm-init/) documentation for self-hosting limitations. -{{< /caution >}} - -{{< tabs name="tab-self-hosting" >}} -{{< tab name="self-hosting" include="generated/kubeadm_alpha_phase_selfhosting_convert-from-staticpods.md" />}} +{{< tabs name="selfhosting" >}} +{{< tab name="selfhosting" include="generated/kubeadm_alpha_selfhosting.md" />}} +{{< tab name="pivot" include="generated/kubeadm_alpha_selfhosting_pivot.md" />}} {{< /tabs >}} ## What's next -* [kubeadm init](/docs/reference/setup-tools/kubeadm/kubeadm-init/) to bootstrap a Kubernetes master node +* [kubeadm init](/docs/reference/setup-tools/kubeadm/kubeadm-init/) to bootstrap a Kubernetes control-plane node * [kubeadm join](/docs/reference/setup-tools/kubeadm/kubeadm-join/) to connect a node to the cluster * [kubeadm reset](/docs/reference/setup-tools/kubeadm/kubeadm-reset/) to revert any changes made to this host by `kubeadm init` or `kubeadm join` diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md index b5e9a19f93..e7ea1eaefe 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md @@ -21,6 +21,10 @@ to print the default configuration and `kubeadm config migrate` to convert your files to a newer version. `kubeadm config images list` and `kubeadm config images pull` can be used to list and pull the images that kubeadm requires. +In Kubernetes v1.13.0 and later to list/pull kube-dns images instead of the CoreDNS image +the `--config` method described [here](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-addon) +has to be used. + {{% /capture %}} {{% capture body %}} @@ -33,8 +37,11 @@ to list and pull the images that kubeadm requires. ## kubeadm config view {#cmd-config-view} {{< include "generated/kubeadm_config_view.md" >}} -## kubeadm config print-default {#cmd-config-print-default} -{{< include "generated/kubeadm_config_print-default.md" >}} +## kubeadm config print init-defaults {#cmd-config-print-init-defaults} +{{< include "generated/kubeadm_config_print_init-defaults.md" >}} + +## kubeadm config print join-defaults {#cmd-config-print-join-defaults} +{{< include "generated/kubeadm_config_print_join-defaults.md" >}} ## kubeadm config migrate {#cmd-config-migrate} {{< include "generated/kubeadm_config_migrate.md" >}} diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md new file mode 100644 index 0000000000..360ac57aac --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md @@ -0,0 +1,155 @@ +--- +title: kubeadm init phase +weight: 90 +--- +In v1.8.0, kubeadm introduced the `kubeadm alpha phase` command with the aim of making kubeadm more modular. In v1.13.0 this command graduated to `kubeadm init phase`. This modularity enables you to invoke atomic sub-steps of the bootstrap process. Hence, you can let kubeadm do some parts and fill in yourself where you need customizations. + +`kubeadm init phase` is consistent with the [kubeadm init workflow](/docs/reference/setup-tools/kubeadm/kubeadm-init/#init-workflow), +and behind the scene both use the same code. + +## kubeadm init phase preflight {#cmd-phase-preflight} + +Using this command you can execute preflight checks on a control-plane node. + +{{< tabs name="tab-preflight" >}} +{{< tab name="preflight" include="generated/kubeadm_init_phase_preflight.md" />}} +{{< /tabs >}} + +## kubeadm init phase certs {#cmd-phase-certs} + +Can be used to create all required certificates by kubeadm. + +{{< tabs name="tab-certs" >}} +{{< tab name="certs" include="generated/kubeadm_init_phase_certs.md" />}} +{{< tab name="all" include="generated/kubeadm_init_phase_certs_all.md" />}} +{{< tab name="apiserver-etcd-client" include="generated/kubeadm_init_phase_certs_apiserver-etcd-client.md" />}} +{{< tab name="apiserver-kubelet-client" include="generated/kubeadm_init_phase_certs_apiserver-kubelet-client.md" />}} +{{< tab name="apiserver" include="generated/kubeadm_init_phase_certs_apiserver.md" />}} +{{< tab name="ca" include="generated/kubeadm_init_phase_certs_ca.md" />}} +{{< tab name="etcd-ca" include="generated/kubeadm_init_phase_certs_etcd-ca.md" />}} +{{< tab name="healthcheck-client" include="generated/kubeadm_init_phase_certs_etcd-healthcheck-client.md" />}} +{{< tab name="etcd-peer" include="generated/kubeadm_init_phase_certs_etcd-peer.md" />}} +{{< tab name="etcd-server" include="generated/kubeadm_init_phase_certs_etcd-server.md" />}} +{{< tab name="front-proxy-ca" include="generated/kubeadm_init_phase_certs_front-proxy-ca.md" />}} +{{< tab name="front-proxy-client" include="generated/kubeadm_init_phase_certs_front-proxy-client.md" />}} +{{< tab name="certs_sa" include="generated/kubeadm_init_phase_certs_sa.md" />}} +{{< /tabs >}} + +## kubeadm init phase kubeconfig {#cmd-phase-kubeconfig} + +You can create all required kubeconfig files by calling the `all` subcommand or call then individually. + +{{< tabs name="tab-kubeconfig" >}} +{{< tab name="kubeconfig" include="generated/kubeadm_init_phase_kubeconfig.md" />}} +{{< tab name="all" include="generated/kubeadm_init_phase_kubeconfig_all.md" />}} +{{< tab name="admin" include="generated/kubeadm_init_phase_kubeconfig_admin.md" />}} +{{< tab name="controller-manager" include="generated/kubeadm_init_phase_kubeconfig_controller-manager.md" />}} +{{< tab name="kubelet" include="generated/kubeadm_init_phase_kubeconfig_kubelet.md" />}} +{{< tab name="scheduler" include="generated/kubeadm_init_phase_kubeconfig_scheduler.md" />}} +{{< /tabs >}} + +## kubeadm init phase kubelet-start {#cmd-phase-kubelet-start} + +This phase will write the kubelet configuration file and environment file and then start the kubelet. + +{{< tabs name="tab-kubelet-start" >}} +{{< tab name="kubelet-start" include="generated/kubeadm_init_phase_kubelet-start.md" />}} +{{< /tabs >}} + +## kubeadm init phase control-plane {#cmd-phase-control-plane} + +Using this phase you can create all required static Pod files for the control plane components. + +{{< tabs name="tab-control-plane" >}} +{{< tab name="control-plane" include="generated/kubeadm_init_phase_control-plane.md" />}} +{{< tab name="all" include="generated/kubeadm_init_phase_control-plane_all.md" />}} +{{< tab name="apiserver" include="generated/kubeadm_init_phase_control-plane_apiserver.md" />}} +{{< tab name="controller-manager" include="generated/kubeadm_init_phase_control-plane_controller-manager.md" />}} +{{< tab name="scheduler" include="generated/kubeadm_init_phase_control-plane_scheduler.md" />}} +{{< /tabs >}} + + +## kubeadm init phase etcd {#cmd-phase-etcd} + +Use the following phase to create a local etcd instance based on a static Pod file. + +{{< tabs name="tab-etcd" >}} +{{< tab name="etcd" include="generated/kubeadm_init_phase_etcd.md" />}} +{{< tab name="local" include="generated/kubeadm_init_phase_etcd_local.md" />}} +{{< /tabs >}} + + +## kubeadm init phase mark-control-plane {#cmd-phase-control-plane} + +Use the following phase to label and taint the node with the `node-role.kubernetes.io/master=""` key-value pair. + +{{< tabs name="tab-mark-control-plane" >}} +{{< tab name="mark-control-plane" include="generated/kubeadm_init_phase_mark-control-plane.md" />}} +{{< /tabs >}} + + +## kubeadm init phase bootstrap-token {#cmd-phase-bootstrap-token} + +Use the following phase to configure bootstrap tokens. + +{{< tabs name="tab-bootstrap-token" >}} +{{< tab name="bootstrap-token" include="generated/kubeadm_init_phase_bootstrap-token.md" />}} +{{< /tabs >}} + + +## kubeadm init phase upload-config {#cmd-phase-upload-config} + +You can use this command to upload the kubeadm configuration to your cluster. +Alternatively, you can use [kubeadm config](/docs/reference/setup-tools/kubeadm/kubeadm-config/). + +{{< tabs name="upload-config" >}} +{{< tab name="upload-config" include="generated/kubeadm_init_phase_upload-config.md" />}} +{{< tab name="all" include="generated/kubeadm_init_phase_upload-config_all.md" />}} +{{< tab name="kubeadm" include="generated/kubeadm_init_phase_upload-config_kubeadm.md" />}} +{{< tab name="kubelet" include="generated/kubeadm_init_phase_upload-config_kubelet.md" />}} +{{< /tabs >}} + + +## kubeadm init phase addon {#cmd-phase-addon} + +You can install all the available addons with the `all` subcommand, or +install them selectively. + +{{< tabs name="tab-addon" >}} +{{< tab name="addon" include="generated/kubeadm_init_phase_addon.md" />}} +{{< tab name="all" include="generated/kubeadm_init_phase_addon_all.md" />}} +{{< tab name="kube-proxy" include="generated/kubeadm_init_phase_addon_kube-proxy.md" />}} +{{< tab name="coredns" include="generated/kubeadm_init_phase_addon_coredns.md" />}} +{{< /tabs >}} + +To use kube-dns instead of CoreDNS you have to pass a configuration file: + +```bash +# for installing a DNS addon only +kubeadm init phase addon coredns --config=someconfig.yaml +# for creating a complete control plane node +kubeadm init --config=someconfig.yaml +# for listing or pulling images +kubeadm config images list/pull --config=someconfig.yaml +# for upgrades +kubeadm upgrade apply --config=someconfig.yaml +``` + +The file has to contain a [`DNS`](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1#DNS) field in[`ClusterConfiguration`](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1#ClusterConfiguration) +and also a type for the addon - `kube-dns` (default value is `CoreDNS`). + +```yaml +apiVersion: kubeadm.k8s.io/v1beta1 +kind: ClusterConfiguration +dns: + type: "kube-dns" +``` + +For more details on each field in the `v1beta1` configuration you can navigate to our +[API reference pages.] (https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1) + +## What's next +* [kubeadm init](/docs/reference/setup-tools/kubeadm/kubeadm-init/) to bootstrap a Kubernetes control-plane node +* [kubeadm join](/docs/reference/setup-tools/kubeadm/kubeadm-join/) to connect a node to the cluster +* [kubeadm reset](/docs/reference/setup-tools/kubeadm/kubeadm-reset/) to revert any changes made to this host by `kubeadm init` or `kubeadm join` +* [kubeadm alpha](/docs/reference/setup-tools/kubeadm/kubeadm-alpha/) to try experimental functionality diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md index ad95ac1e83..72cd8f5a63 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md @@ -8,7 +8,7 @@ content_template: templates/concept weight: 20 --- {{% capture overview %}} -This command initializes a Kubernetes master node. +This command initializes a Kubernetes control-plane node. {{% /capture %}} {{% capture body %}} @@ -16,7 +16,7 @@ This command initializes a Kubernetes master node. {{< include "generated/kubeadm_init.md" >}} ### Init workflow {#init-workflow} -`kubeadm init` bootstraps a Kubernetes master node by executing the +`kubeadm init` bootstraps a Kubernetes control-plane node by executing the following steps: 1. Runs a series of pre-flight checks to validate the system state @@ -36,13 +36,6 @@ following steps: API server, each with its own identity, as well as an additional kubeconfig file for administration named `admin.conf`. -1. If kubeadm is invoked with `--feature-gates=DynamicKubeletConfig` enabled, - it writes the kubelet init configuration into the `/var/lib/kubelet/config/init/kubelet` file. - See [Set Kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file/) - and [Reconfigure a Node's Kubelet in a Live Cluster](/docs/tasks/administer-cluster/reconfigure-kubelet/) - for more information about Dynamic Kubelet Configuration. - This functionality is now by default disabled as it is behind a feature gate, but is expected to be a default in future versions. - 1. Generates static Pod manifests for the API server, controller manager and scheduler. In case an external etcd is not provided, an additional static Pod manifest are generated for etcd. @@ -52,13 +45,7 @@ following steps: Once control plane Pods are up and running, the `kubeadm init` sequence can continue. -1. If kubeadm is invoked with `--feature-gates=DynamicKubeletConfig` enabled, - it completes the kubelet dynamic configuration by creating a ConfigMap and some RBAC rules that enable - kubelets to access to it, and updates the node by pointing `Node.spec.configSource` to the - newly-created ConfigMap. - This functionality is now by default disabled as it is behind a feature gate, but is expected to be a default in future versions. - -1. Apply labels and taints to the master node so that no additional workloads will +1. Apply labels and taints to the control-plane node so that no additional workloads will run there. 1. Generates the token that additional nodes can use to register @@ -82,17 +69,47 @@ following steps: 1. Installs a DNS server (CoreDNS) and the kube-proxy addon components via the API server. In Kubernetes version 1.11 and later CoreDNS is the default DNS server. - To install kube-dns instead of CoreDNS, kubeadm must be invoked with `--feature-gates=CoreDNS=false`. + To install kube-dns instead of CoreDNS, the DNS addon has to configured in the kubeadm `ClusterConfiguration`. For more information about the configuration see the section + `Using kubeadm init with a configuration file` bellow. Please note that although the DNS server is deployed, it will not be scheduled until CNI is installed. -1. If `kubeadm init` is invoked with the alpha self-hosting feature enabled, - (`--feature-gates=SelfHosting=true`), the static Pod based control plane is - transformed into a [self-hosted control plane](#self-hosting). +### Using init phases with kubeadm {#init-phases} + +Kubeadm allows you create a control-plane node in phases. In 1.13 the `kubeadm init phase` command has graduated to GA from it’s previous alpha state under `kubeadm alpha phase`. + +To view the ordered list of phases and sub-phases you can call `kubeadm init --help`. The list will be located at the top of the help screen and each phase will have a description next to it. +Note that by calling `kubeadm init` all of the phases and sub-phases will be executed in this exact order. + +Some phases have unique flags, so if you want to have a look at the list of available options add `--help`, for example: + +```bash +sudo kubeadm init phase control-plane controller-manager --help +``` + +You can also use `--help` to see the list of sub-phases for a certain parent phase: + +```bash +sudo kubeadm init phase control-plane --help +``` + +`kubeadm init` also expose a flag called `--skip-phases` that can be used to skip certain phases. The flag accepts a list of phase names and the names can be taken from the above ordered list. + +An example: + +```bash +sudo kubeadm init phase control-plane all --config=configfile.yaml +sudo kubeadm init phase etcd local --config=configfile.yaml +# you can now modify the control plane and etcd manifest files +sudo kubeadm init --skip-phases=control-plane,etcd --config=configfile.yaml +``` + +What this example would do is write the manifest files for the control plane and etcd in `/etc/kubernetes/manifests` based on the configuration in `configfile.yaml`. This allows you to modify the files and then skip these phases using `--skip-phases`. By calling the last command you will create a control plane node with the custom manifest files. ### Using kubeadm init with a configuration file {#config-file} {{< caution >}} -The config file is still considered alpha and may change in future versions. +**Caution:** The config file is +still considered beta and may change in future versions. {{< /caution >}} It's possible to configure `kubeadm init` with a configuration file instead of command @@ -100,13 +117,13 @@ line flags, and some more advanced features may only be available as configuration file options. This file is passed in the `--config` option. In Kubernetes 1.11 and later, the default configuration can be printed out using the -[kubeadm config print-default](/docs/reference/setup-tools/kubeadm/kubeadm-config/) command. -It is **recommended** that you migrate your old `v1alpha2` configuration to `v1alpha3` using +[kubeadm config print](/docs/reference/setup-tools/kubeadm/kubeadm-config/) command. +It is **recommended** that you migrate your old `v1alpha3` configuration to `v1beta1` using the [kubeadm config migrate](/docs/reference/setup-tools/kubeadm/kubeadm-config/) command, -because `v1alpha2` will be removed in Kubernetes 1.13. +because `v1alpha3` will be removed in Kubernetes 1.14. -For more details on each field in the `v1alpha3` configuration you can navigate to our -[API reference pages.] (https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1alpha3) +For more details on each field in the `v1beta1` configuration you can navigate to our +[API reference pages.] (https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1) ### Adding kube-proxy parameters {#kube-proxy} @@ -266,7 +283,7 @@ In order to set up a cluster where the master and worker nodes communicate with ### Setting the node name -By default, `kubeadm` assigns a node name based on a machine's host address. You can override this setting with the `--node-name`flag. +By default, `kubeadm` assigns a node name based on a machine's host address. You can override this setting with the `--node-name`flag. The flag passes the appropriate [`--hostname-override`](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/#options) to the kubelet. @@ -289,24 +306,15 @@ and will be removed in 1.13. #### Caveats -Self-hosting in 1.8 has some important limitations. In particular, a -self-hosted cluster _cannot recover from a reboot of the master node_ +Self-hosting in 1.8 and later has some important limitations. In particular, a +self-hosted cluster _cannot recover from a reboot of the control-plane node_ without manual intervention. This and other limitations are expected to be resolved before self-hosting graduates from alpha. By default, self-hosted control plane Pods rely on credentials loaded from [`hostPath`](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) volumes. Except for initial creation, these credentials are not managed by -kubeadm. You can use `--feature-gates=StoreCertsInSecrets=true` to enable an -experimental mode where control plane credentials are loaded from Secrets -instead. This requires very careful control over the authentication and -authorization configuration for your cluster, and may not be appropriate for -your environment. - -{{< caution >}} -`StoreCertsInSecrets` is an alpha feature. It is deprecated in 1.12 -and will be removed in 1.13. -{{< /caution >}} +kubeadm. In kubeadm 1.8, the self-hosted portion of the control plane does not include etcd, which still runs as a static Pod. @@ -316,7 +324,7 @@ which still runs as a static Pod. The self-hosting bootstrap process is documented in the [kubeadm design document](https://github.com/kubernetes/kubeadm/blob/master/docs/design/design_v1.9.md#optional-self-hosting). -In summary, `kubeadm init --feature-gates=SelfHosting=true` works as follows: +In summary, `kubeadm alpha selfhosting` works as follows: 1. Waits for this bootstrap static control plane to be running and healthy. This is identical to the `kubeadm init` process without self-hosting. @@ -336,8 +344,6 @@ In summary, `kubeadm init --feature-gates=SelfHosting=true` works as follows: 1. When the original static control plane stops, the new self-hosted control plane is able to bind to listening ports and become active. -This process (steps 3-6) can also be triggered with `kubeadm phase selfhosting convert-from-staticpods`. - ### Running kubeadm without an internet connection For running kubeadm without an internet connection you have to pre-pull the required master images for the version of choice: @@ -388,11 +394,11 @@ know the IP address that the master will have after it is started. kubeadm token generate ``` -1. Start both the master node and the worker nodes concurrently with this token. +1. Start both the control-plane node and the worker nodes concurrently with this token. As they come up they should find each other and form the cluster. The same `--token` argument can be used on both `kubeadm init` and `kubeadm join`. -Once the cluster is up, you can grab the admin credentials from the master node +Once the cluster is up, you can grab the admin credentials from the control-plane node at `/etc/kubernetes/admin.conf` and use that to talk to the cluster. Note that this style of bootstrap has some relaxed security guarantees because @@ -403,6 +409,8 @@ provisioned). For details, see the [kubeadm join](/docs/reference/setup-tools/ku {{% /capture %}} {{% capture whatsnext %}} +* [kubeadm init phase](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/) to understand more about +`kubadm init` phases * [kubeadm join](/docs/reference/setup-tools/kubeadm/kubeadm-join/) to bootstrap a Kubernetes worker node and join it to the cluster * [kubeadm upgrade](/docs/reference/setup-tools/kubeadm/kubeadm-upgrade/) to upgrade a Kubernetes cluster to a newer version * [kubeadm reset](/docs/reference/setup-tools/kubeadm/kubeadm-reset/) to revert any changes made to this host by `kubeadm init` or `kubeadm join` diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-join.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-join.md index 2fb074626b..7cf67a109c 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-join.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-join.md @@ -223,7 +223,7 @@ For details on individual fields in `JoinConfiguration` see [the godoc](https:// {{% /capture %}} {{% capture whatsnext %}} -* [kubeadm init](/docs/reference/setup-tools/kubeadm/kubeadm-init/) to bootstrap a Kubernetes master node +* [kubeadm init](/docs/reference/setup-tools/kubeadm/kubeadm-init/) to bootstrap a Kubernetes control-plane node * [kubeadm token](/docs/reference/setup-tools/kubeadm/kubeadm-token/) to manage tokens for `kubeadm join` * [kubeadm reset](/docs/reference/setup-tools/kubeadm/kubeadm-reset/) to revert any changes made to this host by `kubeadm init` or `kubeadm join` {{% /capture %}} diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md index 611c722666..94ecaeb098 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md @@ -28,6 +28,6 @@ See the [etcd documentation](https://github.com/coreos/etcd/tree/master/etcdctl) {{% /capture %}} {{% capture whatsnext %}} -* [kubeadm init](/docs/reference/setup-tools/kubeadm/kubeadm-init/) to bootstrap a Kubernetes master node +* [kubeadm init](/docs/reference/setup-tools/kubeadm/kubeadm-init/) to bootstrap a Kubernetes control-plane node * [kubeadm join](/docs/reference/setup-tools/kubeadm/kubeadm-join/) to bootstrap a Kubernetes worker node and join it to the cluster {{% /capture %}} diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-token.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-token.md index b4daa61343..0102447ef0 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-token.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-token.md @@ -10,7 +10,7 @@ weight: 70 {{% capture overview %}} Bootstrap tokens are used for establishing bidirectional trust between a node joining -the cluster and a master node, as described in [authenticating with bootstrap tokens](/docs/reference/access-authn-authz/bootstrap-tokens/). +the cluster and a control-plane node, as described in [authenticating with bootstrap tokens](/docs/reference/access-authn-authz/bootstrap-tokens/). `kubeadm init` creates an initial token with a 24-hour TTL. The following commands allow you to manage such a token and also to create and manage new ones. diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-upgrade.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-upgrade.md index 480580aa81..b525a62edc 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-upgrade.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-upgrade.md @@ -21,12 +21,15 @@ For more version-specific upgrade guidance, see the following resources: * [1.10 to 1.11 upgrades](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-11/) * [1.11 to 1.12 upgrades](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-12/) + * [1.12 to 1.13 upgrades](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/) _For older versions, please refer to older documentation sets on the Kubernetes website._ In Kubernetes v1.11.0 and later, you can use `kubeadm upgrade diff` to see the changes that would be applied to static pod manifests. +To use kube-dns with upgrades in Kubernetes v1.13.0 and later please follow [this guide](docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-addon). + ## kubeadm upgrade plan {#cmd-upgrade-plan} {{< include "generated/kubeadm_upgrade_plan.md" >}} diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm.md index c680eb5c40..eccf635588 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm.md @@ -14,7 +14,7 @@ Instead, we expect higher-level and more tailored tooling to be built on top of ## What's next -* [kubeadm init](/docs/reference/setup-tools/kubeadm/kubeadm-init) to bootstrap a Kubernetes master node +* [kubeadm init](/docs/reference/setup-tools/kubeadm/kubeadm-init) to bootstrap a Kubernetes control-plane node * [kubeadm join](/docs/reference/setup-tools/kubeadm/kubeadm-join) to bootstrap a Kubernetes worker node and join it to the cluster * [kubeadm upgrade](/docs/reference/setup-tools/kubeadm/kubeadm-upgrade) to upgrade a Kubernetes cluster to a newer version * [kubeadm config](/docs/reference/setup-tools/kubeadm/kubeadm-config) if you initialized your cluster using kubeadm v1.7.x or lower, to configure your cluster for `kubeadm upgrade` diff --git a/content/en/docs/reference/setup-tools/kubefed/kubefed-init.md b/content/en/docs/reference/setup-tools/kubefed/kubefed-init.md deleted file mode 100644 index 2d5e86d8b6..0000000000 --- a/content/en/docs/reference/setup-tools/kubefed/kubefed-init.md +++ /dev/null @@ -1,105 +0,0 @@ ---- -title: kubefed init -notitle: true -weight: 30 ---- -## kubefed init - -Initialize a federation control plane - -### Synopsis - - -Init initializes a federation control plane. - - Federation control plane is hosted inside a Kubernetes - cluster. The host cluster must be specified using the - --host-cluster-context flag. - -``` -kubefed init FEDERATION_NAME --host-cluster-context=HOST_CONTEXT [flags] -``` - -### Examples - -``` - # Initialize federation control plane for a federation - # named foo in the host cluster whose local kubeconfig - # context is bar. - kubefed init foo --host-cluster-context=bar -``` - -### Options - -``` - --api-server-advertise-address string Preferred address to advertise api server nodeport service. Valid only if 'api-server-service-type=NodePort'. - --api-server-port int32 Preferred port to use for api server nodeport service (0 for random port assignment). Valid only if 'api-server-service-type=NodePort'. - --api-server-service-type string The type of service to create for federation API server. Options: 'LoadBalancer' (default), 'NodePort'. (default "LoadBalancer") - --apiserver-arg-overrides string comma separated list of federation-apiserver arguments to override: Example "--arg1=value1,--arg2=value2..." - --apiserver-enable-basic-auth Enables HTTP Basic authentication for the federation-apiserver. Defaults to false. - --apiserver-enable-token-auth Enables token authentication for the federation-apiserver. Defaults to false. - --controllermanager-arg-overrides string comma separated list of federation-controller-manager arguments to override: Example "--arg1=value1,--arg2=value2..." - --dns-provider string Dns provider to be used for this deployment. - --dns-provider-config string Config file path on local file system for configuring DNS provider. - --dns-zone-name string DNS suffix for this federation. Federated Service DNS names are published with this suffix. - --dry-run dry run without sending commands to server. - --etcd-image string Image to use for etcd server. (default "gcr.io/google_containers/etcd:3.1.10") - --etcd-persistent-storage Use persistent volume for etcd. Defaults to 'true'. (default true) - --etcd-pv-capacity string Size of persistent volume claim to be used for etcd. (default "10Gi") - --etcd-pv-storage-class string The storage class of the persistent volume claim used for etcd. Must be provided if a default storage class is not enabled for the host cluster. - --etcd-servers string External pre-deployed etcd server to be used to store federation state. - --federation-system-namespace string Namespace in the host cluster where the federation system components are installed (default "federation-system") - -h, --help help for init - --host-cluster-context string Host cluster context - --image string Image to use for federation API server and controller manager binaries. (default "gcr.io/k8s-jkns-e2e-gce-federation/fcp-amd64:v0.0.0-master_$Format:%h$") - --image-pull-policy string PullPolicy describes a policy for if/when to pull a container image. The default pull policy is IfNotPresent which will not pull an image if it already exists. (default "IfNotPresent") - --image-pull-secrets string Provide secrets that can access the private registry. - --node-selector string comma separated list of nodeSelector arguments: Example "arg1=value1,arg2=value2..." -``` - -### Options inherited from parent commands - -``` - --alsologtostderr log to standard error as well as files - --as string Username to impersonate for the operation - --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. - --cache-dir string Default HTTP cache directory (default "/Users/jrondeau/.kube/http-cache") - --certificate-authority string Path to a cert file for the certificate authority - --client-certificate string Path to a client certificate file for TLS - --client-key string Path to a client key file for TLS - --cloud-provider-gce-lb-src-cidrs cidrs CIDRs opened in GCE firewall for LB traffic proxy & health checks (default 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16) - --cluster string The name of the kubeconfig cluster to use - --context string The name of the kubeconfig context to use - --default-not-ready-toleration-seconds int Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300) - --default-unreachable-toleration-seconds int Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300) - --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure - --ir-data-source string Data source used by InitialResources. Supported options: influxdb, gcm. (default "influxdb") - --ir-dbname string InfluxDB database name which contains metrics required by InitialResources (default "k8s") - --ir-hawkular string Hawkular configuration URL - --ir-influxdb-host string Address of InfluxDB which contains metrics required by InitialResources (default "localhost:8080/api/v1/namespaces/kube-system/services/monitoring-influxdb:api/proxy") - --ir-namespace-only Whether the estimation should be made only based on data from the same namespace. - --ir-password string Password used for connecting to InfluxDB (default "root") - --ir-percentile int Which percentile of samples should InitialResources use when estimating resources. For experiment purposes. (default 90) - --ir-user string User used for connecting to InfluxDB (default "root") - --kubeconfig string Path to the kubeconfig file to use for CLI requests. - --log-backtrace-at traceLocation when logging hits line file:N, emit a stack trace (default :0) - --log-dir string If non-empty, write log files in this directory - --log-flush-frequency duration Maximum number of seconds between log flushes (default 5s) - --logtostderr log to standard error instead of files (default true) - --match-server-version Require server version to match client version - -n, --namespace string If present, the namespace scope for this CLI request - --password string Password for basic authentication to the API server - --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") - -s, --server string The address and port of the Kubernetes API server - --stderrthreshold severity logs at or above this threshold go to stderr (default 2) - --token string Bearer token for authentication to the API server - --user string The name of the kubeconfig user to use - --username string Username for basic authentication to the API server - -v, --v Level log level for V logs - --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging -``` - -### SEE ALSO -* [kubefed](/docs/reference/setup-tools/kubefed/kubefed/) - kubefed controls a Kubernetes Cluster Federation - -###### Auto generated by spf13/cobra on 25-Mar-2018 diff --git a/content/en/docs/reference/setup-tools/kubefed/kubefed-join.md b/content/en/docs/reference/setup-tools/kubefed/kubefed-join.md deleted file mode 100644 index 25ee8a275d..0000000000 --- a/content/en/docs/reference/setup-tools/kubefed/kubefed-join.md +++ /dev/null @@ -1,99 +0,0 @@ ---- -title: kubefed join -notitle: true -weight: 40 ---- -## kubefed join - -Join a cluster to a federation - -### Synopsis - - -Join adds a cluster to a federation. - - Current context is assumed to be a federation API - server. Please use the --context flag otherwise. - -``` -kubefed join CLUSTER_NAME --host-cluster-context=HOST_CONTEXT [flags] -``` - -### Examples - -``` - # Join a cluster to a federation by specifying the - # cluster name and the context name of the federation - # control plane's host cluster. Cluster name must be - # a valid RFC 1123 subdomain name. Cluster context - # must be specified if the cluster name is different - # than the cluster's context in the local kubeconfig. - kubefed join foo --host-cluster-context=bar -``` - -### Options - -``` - --allow-missing-template-keys If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats. (default true) - --cluster-context string Name of the cluster's context in the local kubeconfig. Defaults to cluster name if unspecified. - --dry-run If true, only print the object that would be sent, without sending it. - --federation-system-namespace string Namespace in the host cluster where the federation system components are installed (default "federation-system") - --generator string The name of the API generator to use. (default "cluster/v1beta1") - -h, --help help for join - --host-cluster-context string Host cluster context - --no-headers When using the default or custom-column output format, don't print headers (default print headers). - -o, --output string Output format. One of: json|yaml|wide|name|custom-columns=...|custom-columns-file=...|go-template=...|go-template-file=...|jsonpath=...|jsonpath-file=... See custom columns [http://kubernetes.io/docs/user-guide/kubectl-overview/#custom-columns], golang template [http://golang.org/pkg/text/template/#pkg-overview] and jsonpath template [http://kubernetes.io/docs/user-guide/jsonpath]. - --save-config If true, the configuration of current object will be saved in its annotation. Otherwise, the annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future. - -a, --show-all When printing, show all resources (default hide terminated pods.) - --show-labels When printing, show all labels as the last column (default hide labels column) - --sort-by string If non-empty, sort list types using this field specification. The field specification is expressed as a JSONPath expression (e.g. '{.metadata.name}'). The field in the API resource specified by this JSONPath expression must be an integer or a string. - --template string Template string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview]. - --validate If true, use a schema to validate the input before sending it (default true) -``` - -### Options inherited from parent commands - -``` - --alsologtostderr log to standard error as well as files - --as string Username to impersonate for the operation - --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. - --cache-dir string Default HTTP cache directory (default "/Users/jrondeau/.kube/http-cache") - --certificate-authority string Path to a cert file for the certificate authority - --client-certificate string Path to a client certificate file for TLS - --client-key string Path to a client key file for TLS - --cloud-provider-gce-lb-src-cidrs cidrs CIDRs opened in GCE firewall for LB traffic proxy & health checks (default 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16) - --cluster string The name of the kubeconfig cluster to use - --context string The name of the kubeconfig context to use - --default-not-ready-toleration-seconds int Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300) - --default-unreachable-toleration-seconds int Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300) - --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure - --ir-data-source string Data source used by InitialResources. Supported options: influxdb, gcm. (default "influxdb") - --ir-dbname string InfluxDB database name which contains metrics required by InitialResources (default "k8s") - --ir-hawkular string Hawkular configuration URL - --ir-influxdb-host string Address of InfluxDB which contains metrics required by InitialResources (default "localhost:8080/api/v1/namespaces/kube-system/services/monitoring-influxdb:api/proxy") - --ir-namespace-only Whether the estimation should be made only based on data from the same namespace. - --ir-password string Password used for connecting to InfluxDB (default "root") - --ir-percentile int Which percentile of samples should InitialResources use when estimating resources. For experiment purposes. (default 90) - --ir-user string User used for connecting to InfluxDB (default "root") - --kubeconfig string Path to the kubeconfig file to use for CLI requests. - --log-backtrace-at traceLocation when logging hits line file:N, emit a stack trace (default :0) - --log-dir string If non-empty, write log files in this directory - --log-flush-frequency duration Maximum number of seconds between log flushes (default 5s) - --logtostderr log to standard error instead of files (default true) - --match-server-version Require server version to match client version - -n, --namespace string If present, the namespace scope for this CLI request - --password string Password for basic authentication to the API server - --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") - -s, --server string The address and port of the Kubernetes API server - --stderrthreshold severity logs at or above this threshold go to stderr (default 2) - --token string Bearer token for authentication to the API server - --user string The name of the kubeconfig user to use - --username string Username for basic authentication to the API server - -v, --v Level log level for V logs - --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging -``` - -### SEE ALSO -* [kubefed](/docs/reference/setup-tools/kubefed/kubefed/) - kubefed controls a Kubernetes Cluster Federation - -###### Auto generated by spf13/cobra on 25-Mar-2018 diff --git a/content/en/docs/reference/setup-tools/kubefed/kubefed-options.md b/content/en/docs/reference/setup-tools/kubefed/kubefed-options.md deleted file mode 100644 index 79b9428013..0000000000 --- a/content/en/docs/reference/setup-tools/kubefed/kubefed-options.md +++ /dev/null @@ -1,77 +0,0 @@ ---- -title: kubefed options -notitle: true -weight: 20 ---- -## kubefed options - -Print the list of flags inherited by all commands - -### Synopsis - - -Print the list of flags inherited by all commands - -``` -kubefed options [flags] -``` - -### Examples - -``` - # Print flags inherited by all commands - kubefed options -``` - -### Options - -``` - -h, --help help for options -``` - -### Options inherited from parent commands - -``` - --alsologtostderr log to standard error as well as files - --as string Username to impersonate for the operation - --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. - --cache-dir string Default HTTP cache directory (default "/Users/jrondeau/.kube/http-cache") - --certificate-authority string Path to a cert file for the certificate authority - --client-certificate string Path to a client certificate file for TLS - --client-key string Path to a client key file for TLS - --cloud-provider-gce-lb-src-cidrs cidrs CIDRs opened in GCE firewall for LB traffic proxy & health checks (default 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16) - --cluster string The name of the kubeconfig cluster to use - --context string The name of the kubeconfig context to use - --default-not-ready-toleration-seconds int Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300) - --default-unreachable-toleration-seconds int Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300) - --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure - --ir-data-source string Data source used by InitialResources. Supported options: influxdb, gcm. (default "influxdb") - --ir-dbname string InfluxDB database name which contains metrics required by InitialResources (default "k8s") - --ir-hawkular string Hawkular configuration URL - --ir-influxdb-host string Address of InfluxDB which contains metrics required by InitialResources (default "localhost:8080/api/v1/namespaces/kube-system/services/monitoring-influxdb:api/proxy") - --ir-namespace-only Whether the estimation should be made only based on data from the same namespace. - --ir-password string Password used for connecting to InfluxDB (default "root") - --ir-percentile int Which percentile of samples should InitialResources use when estimating resources. For experiment purposes. (default 90) - --ir-user string User used for connecting to InfluxDB (default "root") - --kubeconfig string Path to the kubeconfig file to use for CLI requests. - --log-backtrace-at traceLocation when logging hits line file:N, emit a stack trace (default :0) - --log-dir string If non-empty, write log files in this directory - --log-flush-frequency duration Maximum number of seconds between log flushes (default 5s) - --logtostderr log to standard error instead of files (default true) - --match-server-version Require server version to match client version - -n, --namespace string If present, the namespace scope for this CLI request - --password string Password for basic authentication to the API server - --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") - -s, --server string The address and port of the Kubernetes API server - --stderrthreshold severity logs at or above this threshold go to stderr (default 2) - --token string Bearer token for authentication to the API server - --user string The name of the kubeconfig user to use - --username string Username for basic authentication to the API server - -v, --v Level log level for V logs - --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging -``` - -### SEE ALSO -* [kubefed](/docs/reference/setup-tools/kubefed/kubefed/) - kubefed controls a Kubernetes Cluster Federation - -###### Auto generated by spf13/cobra on 25-Mar-2018 diff --git a/content/en/docs/reference/setup-tools/kubefed/kubefed-unjoin.md b/content/en/docs/reference/setup-tools/kubefed/kubefed-unjoin.md deleted file mode 100644 index 2239196832..0000000000 --- a/content/en/docs/reference/setup-tools/kubefed/kubefed-unjoin.md +++ /dev/null @@ -1,86 +0,0 @@ ---- -title: kubefed unjoin -notitle: true -weight: 50 ---- -## kubefed unjoin - -Unjoin a cluster from a federation - -### Synopsis - - -Unjoin a cluster from a federation. - - Current context is assumed to be a federation endpoint. - Please use the --context flag otherwise. - -``` -kubefed unjoin CLUSTER_NAME --host-cluster-context=HOST_CONTEXT [flags] -``` - -### Examples - -``` - # Unjoin the specified cluster from a federation. - # Federation control plane's host cluster context name - # must be specified via the --host-cluster-context flag - # to properly cleanup the credentials. - kubectl unjoin foo --host-cluster-context=bar --cluster-context=baz -``` - -### Options - -``` - --cluster-context string Name of the cluster's context in the local kubeconfig. Defaults to cluster name if unspecified. - --federation-system-namespace string Namespace in the host cluster where the federation system components are installed (default "federation-system") - -h, --help help for unjoin - --host-cluster-context string Host cluster context -``` - -### Options inherited from parent commands - -``` - --alsologtostderr log to standard error as well as files - --as string Username to impersonate for the operation - --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. - --cache-dir string Default HTTP cache directory (default "/Users/jrondeau/.kube/http-cache") - --certificate-authority string Path to a cert file for the certificate authority - --client-certificate string Path to a client certificate file for TLS - --client-key string Path to a client key file for TLS - --cloud-provider-gce-lb-src-cidrs cidrs CIDRs opened in GCE firewall for LB traffic proxy & health checks (default 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16) - --cluster string The name of the kubeconfig cluster to use - --context string The name of the kubeconfig context to use - --default-not-ready-toleration-seconds int Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300) - --default-unreachable-toleration-seconds int Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300) - --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure - --ir-data-source string Data source used by InitialResources. Supported options: influxdb, gcm. (default "influxdb") - --ir-dbname string InfluxDB database name which contains metrics required by InitialResources (default "k8s") - --ir-hawkular string Hawkular configuration URL - --ir-influxdb-host string Address of InfluxDB which contains metrics required by InitialResources (default "localhost:8080/api/v1/namespaces/kube-system/services/monitoring-influxdb:api/proxy") - --ir-namespace-only Whether the estimation should be made only based on data from the same namespace. - --ir-password string Password used for connecting to InfluxDB (default "root") - --ir-percentile int Which percentile of samples should InitialResources use when estimating resources. For experiment purposes. (default 90) - --ir-user string User used for connecting to InfluxDB (default "root") - --kubeconfig string Path to the kubeconfig file to use for CLI requests. - --log-backtrace-at traceLocation when logging hits line file:N, emit a stack trace (default :0) - --log-dir string If non-empty, write log files in this directory - --log-flush-frequency duration Maximum number of seconds between log flushes (default 5s) - --logtostderr log to standard error instead of files (default true) - --match-server-version Require server version to match client version - -n, --namespace string If present, the namespace scope for this CLI request - --password string Password for basic authentication to the API server - --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") - -s, --server string The address and port of the Kubernetes API server - --stderrthreshold severity logs at or above this threshold go to stderr (default 2) - --token string Bearer token for authentication to the API server - --user string The name of the kubeconfig user to use - --username string Username for basic authentication to the API server - -v, --v Level log level for V logs - --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging -``` - -### SEE ALSO -* [kubefed](/docs/reference/setup-tools/kubefed/kubefed/) - kubefed controls a Kubernetes Cluster Federation - -###### Auto generated by spf13/cobra on 25-Mar-2018 diff --git a/content/en/docs/reference/setup-tools/kubefed/kubefed-version.md b/content/en/docs/reference/setup-tools/kubefed/kubefed-version.md deleted file mode 100644 index d605404fc0..0000000000 --- a/content/en/docs/reference/setup-tools/kubefed/kubefed-version.md +++ /dev/null @@ -1,80 +0,0 @@ ---- -title: kubefed version -notitle: true -weight: 60 ---- -## kubefed version - -Print the client and server version information - -### Synopsis - - -Print the client and server version information for the current context - -``` -kubefed version [flags] -``` - -### Examples - -``` - # Print the client and server versions for the current context - kubefed version -``` - -### Options - -``` - --client Client version only (no server required). - -h, --help help for version - -o, --output string One of 'yaml' or 'json'. - --short Print just the version number. -``` - -### Options inherited from parent commands - -``` - --alsologtostderr log to standard error as well as files - --as string Username to impersonate for the operation - --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. - --cache-dir string Default HTTP cache directory (default "/Users/jrondeau/.kube/http-cache") - --certificate-authority string Path to a cert file for the certificate authority - --client-certificate string Path to a client certificate file for TLS - --client-key string Path to a client key file for TLS - --cloud-provider-gce-lb-src-cidrs cidrs CIDRs opened in GCE firewall for LB traffic proxy & health checks (default 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16) - --cluster string The name of the kubeconfig cluster to use - --context string The name of the kubeconfig context to use - --default-not-ready-toleration-seconds int Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300) - --default-unreachable-toleration-seconds int Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300) - --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure - --ir-data-source string Data source used by InitialResources. Supported options: influxdb, gcm. (default "influxdb") - --ir-dbname string InfluxDB database name which contains metrics required by InitialResources (default "k8s") - --ir-hawkular string Hawkular configuration URL - --ir-influxdb-host string Address of InfluxDB which contains metrics required by InitialResources (default "localhost:8080/api/v1/namespaces/kube-system/services/monitoring-influxdb:api/proxy") - --ir-namespace-only Whether the estimation should be made only based on data from the same namespace. - --ir-password string Password used for connecting to InfluxDB (default "root") - --ir-percentile int Which percentile of samples should InitialResources use when estimating resources. For experiment purposes. (default 90) - --ir-user string User used for connecting to InfluxDB (default "root") - --kubeconfig string Path to the kubeconfig file to use for CLI requests. - --log-backtrace-at traceLocation when logging hits line file:N, emit a stack trace (default :0) - --log-dir string If non-empty, write log files in this directory - --log-flush-frequency duration Maximum number of seconds between log flushes (default 5s) - --logtostderr log to standard error instead of files (default true) - --match-server-version Require server version to match client version - -n, --namespace string If present, the namespace scope for this CLI request - --password string Password for basic authentication to the API server - --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") - -s, --server string The address and port of the Kubernetes API server - --stderrthreshold severity logs at or above this threshold go to stderr (default 2) - --token string Bearer token for authentication to the API server - --user string The name of the kubeconfig user to use - --username string Username for basic authentication to the API server - -v, --v Level log level for V logs - --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging -``` - -### SEE ALSO -* [kubefed](/docs/reference/setup-tools/kubefed/kubefed/) - kubefed controls a Kubernetes Cluster Federation - -###### Auto generated by spf13/cobra on 25-Mar-2018 diff --git a/content/en/docs/reference/setup-tools/kubefed/kubefed.md b/content/en/docs/reference/setup-tools/kubefed/kubefed.md index 4d296ba22c..8e424f4abe 100644 --- a/content/en/docs/reference/setup-tools/kubefed/kubefed.md +++ b/content/en/docs/reference/setup-tools/kubefed/kubefed.md @@ -1,3 +1,9 @@ +--- +title: kubefed +notitle: true +weight: 10 +--- + ## kubefed kubefed controls a Kubernetes Cluster Federation @@ -19,7 +25,7 @@ kubefed [flags] --alsologtostderr log to standard error as well as files --as string Username to impersonate for the operation --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. - --cache-dir string Default HTTP cache directory (default "/Users/zarnold/.kube/http-cache") + --cache-dir string Default HTTP cache directory (default "/Users/tim/.kube/http-cache") --certificate-authority string Path to a cert file for the certificate authority --client-certificate string Path to a client certificate file for TLS --client-key string Path to a client key file for TLS @@ -63,4 +69,4 @@ kubefed [flags] * [kubefed unjoin](kubefed_unjoin.md) - Unjoin a cluster from a federation * [kubefed version](kubefed_version.md) - Print the client and server version information -###### Auto generated by spf13/cobra on 24-Sep-2018 +###### Auto generated by spf13/cobra on 1-Dec-2018 diff --git a/content/en/docs/reference/setup-tools/kubefed/kubefed_init.md b/content/en/docs/reference/setup-tools/kubefed/kubefed_init.md index 69da8c83fe..1748b5b3d3 100644 --- a/content/en/docs/reference/setup-tools/kubefed/kubefed_init.md +++ b/content/en/docs/reference/setup-tools/kubefed/kubefed_init.md @@ -1,3 +1,9 @@ +--- +title: kubefed init +notitle: true +weight: 30 +--- + ## kubefed init Initialize a federation control plane @@ -59,7 +65,7 @@ kubefed init FEDERATION_NAME --host-cluster-context=HOST_CONTEXT [flags] --alsologtostderr log to standard error as well as files --as string Username to impersonate for the operation --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. - --cache-dir string Default HTTP cache directory (default "/Users/zarnold/.kube/http-cache") + --cache-dir string Default HTTP cache directory (default "/Users/tim/.kube/http-cache") --certificate-authority string Path to a cert file for the certificate authority --client-certificate string Path to a client certificate file for TLS --client-key string Path to a client key file for TLS @@ -98,4 +104,4 @@ kubefed init FEDERATION_NAME --host-cluster-context=HOST_CONTEXT [flags] ### SEE ALSO * [kubefed](kubefed.md) - kubefed controls a Kubernetes Cluster Federation -###### Auto generated by spf13/cobra on 24-Sep-2018 +###### Auto generated by spf13/cobra on 1-Dec-2018 diff --git a/content/en/docs/reference/setup-tools/kubefed/kubefed_join.md b/content/en/docs/reference/setup-tools/kubefed/kubefed_join.md index 79b29a4f99..c44520a7b8 100644 --- a/content/en/docs/reference/setup-tools/kubefed/kubefed_join.md +++ b/content/en/docs/reference/setup-tools/kubefed/kubefed_join.md @@ -1,3 +1,9 @@ +--- +title: kubefed join +notitle: true +weight: 40 +--- + ## kubefed join Join a cluster to a federation @@ -53,7 +59,7 @@ kubefed join CLUSTER_NAME --host-cluster-context=HOST_CONTEXT [flags] --alsologtostderr log to standard error as well as files --as string Username to impersonate for the operation --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. - --cache-dir string Default HTTP cache directory (default "/Users/zarnold/.kube/http-cache") + --cache-dir string Default HTTP cache directory (default "/Users/tim/.kube/http-cache") --certificate-authority string Path to a cert file for the certificate authority --client-certificate string Path to a client certificate file for TLS --client-key string Path to a client key file for TLS @@ -92,4 +98,4 @@ kubefed join CLUSTER_NAME --host-cluster-context=HOST_CONTEXT [flags] ### SEE ALSO * [kubefed](kubefed.md) - kubefed controls a Kubernetes Cluster Federation -###### Auto generated by spf13/cobra on 24-Sep-2018 +###### Auto generated by spf13/cobra on 1-Dec-2018 diff --git a/content/en/docs/reference/setup-tools/kubefed/kubefed_options.md b/content/en/docs/reference/setup-tools/kubefed/kubefed_options.md index d1255f2865..7e9bca651d 100644 --- a/content/en/docs/reference/setup-tools/kubefed/kubefed_options.md +++ b/content/en/docs/reference/setup-tools/kubefed/kubefed_options.md @@ -1,3 +1,9 @@ +--- +title: kubefed options +notitle: true +weight: 20 +--- + ## kubefed options Print the list of flags inherited by all commands @@ -30,7 +36,7 @@ kubefed options [flags] --alsologtostderr log to standard error as well as files --as string Username to impersonate for the operation --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. - --cache-dir string Default HTTP cache directory (default "/Users/zarnold/.kube/http-cache") + --cache-dir string Default HTTP cache directory (default "/Users/tim/.kube/http-cache") --certificate-authority string Path to a cert file for the certificate authority --client-certificate string Path to a client certificate file for TLS --client-key string Path to a client key file for TLS @@ -69,4 +75,4 @@ kubefed options [flags] ### SEE ALSO * [kubefed](kubefed.md) - kubefed controls a Kubernetes Cluster Federation -###### Auto generated by spf13/cobra on 24-Sep-2018 +###### Auto generated by spf13/cobra on 1-Dec-2018 diff --git a/content/en/docs/reference/setup-tools/kubefed/kubefed_unjoin.md b/content/en/docs/reference/setup-tools/kubefed/kubefed_unjoin.md index ae80ef3bb8..c2f8bb3e95 100644 --- a/content/en/docs/reference/setup-tools/kubefed/kubefed_unjoin.md +++ b/content/en/docs/reference/setup-tools/kubefed/kubefed_unjoin.md @@ -1,3 +1,9 @@ +--- +title: kubefed unjoin +notitle: true +weight: 50 +--- + ## kubefed unjoin Unjoin a cluster from a federation @@ -40,7 +46,7 @@ kubefed unjoin CLUSTER_NAME --host-cluster-context=HOST_CONTEXT [flags] --alsologtostderr log to standard error as well as files --as string Username to impersonate for the operation --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. - --cache-dir string Default HTTP cache directory (default "/Users/zarnold/.kube/http-cache") + --cache-dir string Default HTTP cache directory (default "/Users/tim/.kube/http-cache") --certificate-authority string Path to a cert file for the certificate authority --client-certificate string Path to a client certificate file for TLS --client-key string Path to a client key file for TLS @@ -79,4 +85,4 @@ kubefed unjoin CLUSTER_NAME --host-cluster-context=HOST_CONTEXT [flags] ### SEE ALSO * [kubefed](kubefed.md) - kubefed controls a Kubernetes Cluster Federation -###### Auto generated by spf13/cobra on 24-Sep-2018 +###### Auto generated by spf13/cobra on 1-Dec-2018 diff --git a/content/en/docs/reference/setup-tools/kubefed/kubefed_version.md b/content/en/docs/reference/setup-tools/kubefed/kubefed_version.md index eb2cfdbfff..f9343bd7fe 100644 --- a/content/en/docs/reference/setup-tools/kubefed/kubefed_version.md +++ b/content/en/docs/reference/setup-tools/kubefed/kubefed_version.md @@ -1,3 +1,9 @@ +--- +title: kubefed version +notitle: true +weight: 60 +--- + ## kubefed version Print the client and server version information @@ -33,7 +39,7 @@ kubefed version [flags] --alsologtostderr log to standard error as well as files --as string Username to impersonate for the operation --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. - --cache-dir string Default HTTP cache directory (default "/Users/zarnold/.kube/http-cache") + --cache-dir string Default HTTP cache directory (default "/Users/tim/.kube/http-cache") --certificate-authority string Path to a cert file for the certificate authority --client-certificate string Path to a client certificate file for TLS --client-key string Path to a client key file for TLS @@ -72,4 +78,4 @@ kubefed version [flags] ### SEE ALSO * [kubefed](kubefed.md) - kubefed controls a Kubernetes Cluster Federation -###### Auto generated by spf13/cobra on 24-Sep-2018 +###### Auto generated by spf13/cobra on 1-Dec-2018 diff --git a/content/en/docs/reference/using-api/api-concepts.md b/content/en/docs/reference/using-api/api-concepts.md index 1ac0712f47..5438cd2a7d 100644 --- a/content/en/docs/reference/using-api/api-concepts.md +++ b/content/en/docs/reference/using-api/api-concepts.md @@ -86,7 +86,7 @@ For example: } ... -A given Kubernetes server will only preserve a historical list of changes for a limited time. Older clusters using etcd2 preserve a maximum of 1000 changes. Newer clusters using etcd3 preserve changes in the last 5 minutes by default. When the requested watch operations fail because the historical version of that resource is not available, clients must handle the case by recognizing the status code `410 Gone`, clearing their local cache, performing a list operation, and starting the watch from the `resourceVersion` returned by that new list operation. Most client libraries offer some form of standard tool for this logic. (In Go this is called a `Reflector` and is located in the `k8s.io/client-go/cache` package.) +A given Kubernetes server will only preserve a historical list of changes for a limited time. Clusters using etcd3 preserve changes in the last 5 minutes by default. When the requested watch operations fail because the historical version of that resource is not available, clients must handle the case by recognizing the status code `410 Gone`, clearing their local cache, performing a list operation, and starting the watch from the `resourceVersion` returned by that new list operation. Most client libraries offer some form of standard tool for this logic. (In Go this is called a `Reflector` and is located in the `k8s.io/client-go/cache` package.) ## Retrieving large results sets in chunks @@ -287,26 +287,14 @@ Clients that receive a response in `application/vnd.kubernetes.protobuf` that do ## Dry run -{{< feature-state for_k8s_version="v1.12" state="alpha" >}} In version 1.12, if the dry run alpha feature is enabled, the modifying verbs (`POST`, `PUT`, `PATCH`, and `DELETE`) can accept requests in a dry run mode. Dry run mode helps to evaluate a request through the typical request stages (admission chain, validation, merge conflicts) up until persisting objects to storage. The response body for the request is as close as possible to a non dry run response. The system guarantees that dry run requests will not be persisted in storage or have any other side effects. - - -### Enable the dry run alpha feature - -Dry run is an alpha feature, so it is disabled by default. To turn it on, -you need to: - -* Include "DryRun=true" in the `--feature-gates` flag when starting - `kube-apiserver`. If you have multiple `kube-apiserver` replicas, all should - have the same flag setting. - -If this feature is not enabled, all requests with a modifying verb (`POST`, `PUT`, `PATCH`, and `DELETE`) which set the `dryRun` query parameter will be rejected with a 400 Bad Request error. Kubernetes 1.11 always rejects dry run requests like this, so it is safe for clients to make dry run requests even if the feature is not enabled on the server, as long as the server version is >= 1.11. +{{< feature-state for_k8s_version="v1.13" state="beta" >}} In version 1.13, the dry run beta feature is enabled by default. The modifying verbs (`POST`, `PUT`, `PATCH`, and `DELETE`) can accept requests in a dry run mode. Dry run mode helps to evaluate a request through the typical request stages (admission chain, validation, merge conflicts) up until persisting objects to storage. The response body for the request is as close as possible to a non dry run response. The system guarantees that dry run requests will not be persisted in storage or have any other side effects. ### Make a dry run request -Dry run is triggered by setting the `dryRun` query parameter. This parameter is a string, working as an enum, and in 1.12 the only accepted values are: +Dry run is triggered by setting the `dryRun` query parameter. This parameter is a string, working as an enum, and in 1.13 the only accepted values are: -* `All`: Every stage runs as normal, except for the final storage stage. Admission controllers are run to check that the request is valid, mutating controllers mutate the request, merge is performed on `PATCH`, fields are defaulted, and schema validation occurs. The changes are not persisted to the underlying storage, but the final object which would have been persisted is still returned to the user, along with the normal status code. If the request would trigger an admission controller which would have side effects, the request will be failed rather than risk an unwanted side effect. Admission webhooks can now declare (in their configuration object) that they do not have side effects to prevent this. All built in admission control plugins support dry run. +* `All`: Every stage runs as normal, except for the final storage stage. Admission controllers are run to check that the request is valid, mutating controllers mutate the request, merge is performed on `PATCH`, fields are defaulted, and schema validation occurs. The changes are not persisted to the underlying storage, but the final object which would have been persisted is still returned to the user, along with the normal status code. If the request would trigger an admission controller which would have side effects, the request will be failed rather than risk an unwanted side effect. All built in admission control plugins support dry run. Additionally, admission webhooks can declare in their [configuration object](/docs/reference/generated/kubernetes-api/v1.13/#webhook-v1beta1-admissionregistration-k8s-io) that they do not have side effects by setting the sideEffects field to "None". If a webhook actually does have side effects, then the sideEffects field should be set to "NoneOnDryRun", and the webhook should also be modified to understand the `DryRun` field in AdmissionReview, and prevent side effects on dry run requests. * Leave the value empty, which is also the default: Keep the default modifying behavior. For example: diff --git a/content/en/docs/setup/custom-cloud/master.yaml b/content/en/docs/setup/custom-cloud/master.yaml deleted file mode 100644 index 5b7df1bd77..0000000000 --- a/content/en/docs/setup/custom-cloud/master.yaml +++ /dev/null @@ -1,142 +0,0 @@ -#cloud-config - ---- -write-files: - - path: /etc/conf.d/nfs - permissions: '0644' - content: | - OPTS_RPC_MOUNTD="" - - path: /opt/bin/wupiao - permissions: '0755' - content: | - #!/bin/bash - # [w]ait [u]ntil [p]ort [i]s [a]ctually [o]pen - [ -n "$1" ] && \ - until curl -o /dev/null -sIf http://${1}; do \ - sleep 1 && echo .; - done; - exit $? - -hostname: master -coreos: - etcd2: - name: master - listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001 - advertise-client-urls: http://$private_ipv4:2379,http://$private_ipv4:4001 - initial-cluster-token: k8s_etcd - listen-peer-urls: http://$private_ipv4:2380,http://$private_ipv4:7001 - initial-advertise-peer-urls: http://$private_ipv4:2380 - initial-cluster: master=http://$private_ipv4:2380 - initial-cluster-state: new - fleet: - metadata: "role=master" - units: - - name: etcd2.service - command: start - - name: generate-serviceaccount-key.service - command: start - content: | - [Unit] - Description=Generate service-account key file - - [Service] - ExecStartPre=-/usr/bin/mkdir -p /opt/bin - ExecStart=/bin/openssl genrsa -out /opt/bin/kube-serviceaccount.key 2048 2>/dev/null - RemainAfterExit=yes - Type=oneshot - - name: setup-network-environment.service - command: start - content: | - [Unit] - Description=Setup Network Environment - Documentation=https://github.com/kelseyhightower/setup-network-environment - Requires=network-online.target - After=network-online.target - - [Service] - ExecStartPre=-/usr/bin/mkdir -p /opt/bin - ExecStartPre=/usr/bin/curl -L -o /opt/bin/setup-network-environment -z /opt/bin/setup-network-environment https://github.com/kelseyhightower/setup-network-environment/releases/download/v1.0.0/setup-network-environment - ExecStartPre=/usr/bin/chmod +x /opt/bin/setup-network-environment - ExecStart=/opt/bin/setup-network-environment - RemainAfterExit=yes - Type=oneshot - - name: fleet.service - command: start - - name: flanneld.service - command: start - drop-ins: - - name: 50-network-config.conf - content: | - [Unit] - Requires=etcd2.service - [Service] - ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.244.0.0/16", "Backend": {"Type": "vxlan"}}' - - name: docker.service - command: start - - name: kube-apiserver.service - command: start - content: | - [Unit] - Description=Kubernetes API Server - Documentation=https://github.com/kubernetes/kubernetes - Requires=setup-network-environment.service etcd2.service generate-serviceaccount-key.service - After=setup-network-environment.service etcd2.service generate-serviceaccount-key.service - - [Service] - EnvironmentFile=/etc/network-environment - ExecStartPre=-/usr/bin/mkdir -p /opt/bin - ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-apiserver -z /opt/bin/kube-apiserver https://storage.googleapis.com/kubernetes-release/release/v1.1.2/bin/linux/amd64/kube-apiserver - ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-apiserver - ExecStartPre=/opt/bin/wupiao 127.0.0.1:2379/v2/machines - ExecStart=/opt/bin/kube-apiserver \ - --service-account-key-file=/opt/bin/kube-serviceaccount.key \ - --service-account-lookup=false \ - --enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota \ - --runtime-config=api/v1 \ - --allow-privileged=true \ - --insecure-bind-address=0.0.0.0 \ - --insecure-port=8080 \ - --kubelet-https=true \ - --secure-port=6443 \ - --service-cluster-ip-range=10.100.0.0/16 \ - --etcd-servers=http://127.0.0.1:2379 \ - --public-address-override=${DEFAULT_IPV4} \ - --logtostderr=true - Restart=always - RestartSec=10 - - name: kube-controller-manager.service - command: start - content: | - [Unit] - Description=Kubernetes Controller Manager - Documentation=https://github.com/kubernetes/kubernetes - Requires=kube-apiserver.service - After=kube-apiserver.service - - [Service] - ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-controller-manager -z /opt/bin/kube-controller-manager https://storage.googleapis.com/kubernetes-release/release/v1.1.2/bin/linux/amd64/kube-controller-manager - ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-controller-manager - ExecStart=/opt/bin/kube-controller-manager \ - --service-account-private-key-file=/opt/bin/kube-serviceaccount.key \ - --master=127.0.0.1:8080 \ - --logtostderr=true - Restart=always - RestartSec=10 - - name: kube-scheduler.service - command: start - content: | - [Unit] - Description=Kubernetes Scheduler - Documentation=https://github.com/kubernetes/kubernetes - Requires=kube-apiserver.service - After=kube-apiserver.service - - [Service] - ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-scheduler -z /opt/bin/kube-scheduler https://storage.googleapis.com/kubernetes-release/release/v1.1.2/bin/linux/amd64/kube-scheduler - ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-scheduler - ExecStart=/opt/bin/kube-scheduler --master=127.0.0.1:8080 - Restart=always - RestartSec=10 - update: - group: alpha - reboot-strategy: off diff --git a/content/en/docs/setup/custom-cloud/node.yaml b/content/en/docs/setup/custom-cloud/node.yaml deleted file mode 100644 index 9f5caff49b..0000000000 --- a/content/en/docs/setup/custom-cloud/node.yaml +++ /dev/null @@ -1,92 +0,0 @@ -#cloud-config -write-files: - - path: /opt/bin/wupiao - permissions: '0755' - content: | - #!/bin/bash - # [w]ait [u]ntil [p]ort [i]s [a]ctually [o]pen - [ -n "$1" ] && [ -n "$2" ] && while ! curl --output /dev/null \ - --silent --head --fail \ - http://${1}:${2}; do sleep 1 && echo -n .; done; - exit $? -coreos: - etcd2: - listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001 - advertise-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001 - initial-cluster: master=http://:2380 - proxy: on - fleet: - metadata: "role=node" - units: - - name: etcd2.service - command: start - - name: fleet.service - command: start - - name: flanneld.service - command: start - - name: docker.service - command: start - - name: setup-network-environment.service - command: start - content: | - [Unit] - Description=Setup Network Environment - Documentation=https://github.com/kelseyhightower/setup-network-environment - Requires=network-online.target - After=network-online.target - - [Service] - ExecStartPre=-/usr/bin/mkdir -p /opt/bin - ExecStartPre=/usr/bin/curl -L -o /opt/bin/setup-network-environment -z /opt/bin/setup-network-environment https://github.com/kelseyhightower/setup-network-environment/releases/download/v1.0.0/setup-network-environment - ExecStartPre=/usr/bin/chmod +x /opt/bin/setup-network-environment - ExecStart=/opt/bin/setup-network-environment - RemainAfterExit=yes - Type=oneshot - - name: kube-proxy.service - command: start - content: | - [Unit] - Description=Kubernetes Proxy - Documentation=https://github.com/kubernetes/kubernetes - Requires=setup-network-environment.service - After=setup-network-environment.service - - [Service] - ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-proxy -z /opt/bin/kube-proxy https://storage.googleapis.com/kubernetes-release/release/v1.1.2/bin/linux/amd64/kube-proxy - ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-proxy - # wait for kubernetes master to be up and ready - ExecStartPre=/opt/bin/wupiao 8080 - ExecStart=/opt/bin/kube-proxy \ - --master=:8080 \ - --logtostderr=true - Restart=always - RestartSec=10 - - name: kube-kubelet.service - command: start - content: | - [Unit] - Description=Kubernetes Kubelet - Documentation=https://github.com/kubernetes/kubernetes - Requires=setup-network-environment.service - After=setup-network-environment.service - - [Service] - EnvironmentFile=/etc/network-environment - ExecStartPre=/usr/bin/curl -L -o /opt/bin/kubelet -z /opt/bin/kubelet https://storage.googleapis.com/kubernetes-release/release/v1.1.2/bin/linux/amd64/kubelet - ExecStartPre=/usr/bin/chmod +x /opt/bin/kubelet - # wait for kubernetes master to be up and ready - ExecStartPre=/opt/bin/wupiao 8080 - ExecStart=/opt/bin/kubelet \ - --address=0.0.0.0 \ - --port=10250 \ - --hostname-override=${DEFAULT_IPV4} \ - --api-servers=:8080 \ - --allow-privileged=true \ - --logtostderr=true \ - --healthz-bind-address=0.0.0.0 \ - --healthz-port=10248 - Restart=always - RestartSec=10 - update: - group: alpha - reboot-strategy: off diff --git a/content/en/docs/setup/independent/control-plane-flags.md b/content/en/docs/setup/independent/control-plane-flags.md index eb40dcfa58..1929b1ae17 100644 --- a/content/en/docs/setup/independent/control-plane-flags.md +++ b/content/en/docs/setup/independent/control-plane-flags.md @@ -8,15 +8,16 @@ weight: 40 {{% capture overview %}} -The kubeadm configuration exposes the following fields that can override the default flags passed to control plane components such as the APIServer, ControllerManager and Scheduler: +The kubeadm `ClusterConfiguration` object exposes the field `extraArgs` that can override the default flags passed to control plane +components such as the APIServer, ControllerManager and Scheduler. The components are defined using the following fields: -- `APIServerExtraArgs` -- `ControllerManagerExtraArgs` -- `SchedulerExtraArgs` +- `apiServer` +- `controllerManager` +- `scheduler` -These fields consist of `key: value` pairs. To override a flag for a control plane component: +The `extraArgs` field consist of `key: value` pairs. To override a flag for a control plane component: -1. Add the appropriate field to your configuration. +1. Add the appropriate fields to your configuration. 2. Add the flags to override to the field. For more details on each field in the configuration you can navigate to our @@ -32,16 +33,17 @@ For details, see the [reference documentation for kube-apiserver](/docs/referenc Example usage: ```yaml -apiVersion: kubeadm.k8s.io/v1alpha3 +apiVersion: kubeadm.k8s.io/v1beta1 kind: ClusterConfiguration -kubernetesVersion: v1.12.0 +kubernetesVersion: v1.13.0 metadata: - name: 1.12-sample -apiServerExtraArgs: - advertise-address: 192.168.0.103 - anonymous-auth: false - enable-admission-plugins: AlwaysPullImages,DefaultStorageClass - audit-log-path: /home/johndoe/audit.log + name: 1.13-sample +apiServer: + extraArgs: + advertise-address: 192.168.0.103 + anonymous-auth: false + enable-admission-plugins: AlwaysPullImages,DefaultStorageClass + audit-log-path: /home/johndoe/audit.log ``` ## ControllerManager flags @@ -50,15 +52,16 @@ For details, see the [reference documentation for kube-controller-manager](/docs Example usage: ```yaml -apiVersion: kubeadm.k8s.io/v1alpha3 +apiVersion: kubeadm.k8s.io/v1beta1 kind: ClusterConfiguration -kubernetesVersion: v1.12.0 +kubernetesVersion: v1.13.0 metadata: - name: 1.12-sample -controllerManagerExtraArgs: - cluster-signing-key-file: /home/johndoe/keys/ca.key - bind-address: 0.0.0.0 - deployment-controller-sync-period: 50 + name: 1.13-sample +controllerManager: + extraArgs: + cluster-signing-key-file: /home/johndoe/keys/ca.key + bind-address: 0.0.0.0 + deployment-controller-sync-period: 50 ``` ## Scheduler flags @@ -67,15 +70,16 @@ For details, see the [reference documentation for kube-scheduler](/docs/referenc Example usage: ```yaml -apiVersion: kubeadm.k8s.io/v1alpha3 +apiVersion: kubeadm.k8s.io/v1beta1 kind: ClusterConfiguration -kubernetesVersion: v1.12.0 +kubernetesVersion: v1.13.0 metadata: - name: 1.12-sample -schedulerExtraArgs: - address: 0.0.0.0 - config: /home/johndoe/schedconfig.yaml - kubeconfig: /home/johndoe/kubeconfig.yaml + name: 1.13-sample +scheduler: + extraArgs: + address: 0.0.0.0 + config: /home/johndoe/schedconfig.yaml + kubeconfig: /home/johndoe/kubeconfig.yaml ``` {{% /capture %}} diff --git a/content/en/docs/setup/independent/create-cluster-kubeadm.md b/content/en/docs/setup/independent/create-cluster-kubeadm.md index 8db68b4f83..c69ec05132 100644 --- a/content/en/docs/setup/independent/create-cluster-kubeadm.md +++ b/content/en/docs/setup/independent/create-cluster-kubeadm.md @@ -37,21 +37,20 @@ but you may also build them from source for other OSes. | Area | Maturity Level | |---------------------------|--------------- | -| Command line UX | beta | -| Implementation | beta | -| Config file API | alpha | -| Self-hosting | alpha | -| kubeadm alpha subcommands | alpha | +| Command line UX | GA | +| Implementation | GA | +| Config file API | beta | | CoreDNS | GA | +| kubeadm alpha subcommands | alpha | +| High availability | alpha | | DynamicKubeletConfig | alpha | +| Self-hosting | alpha | -kubeadm's overall feature state is **Beta** and will soon be graduated to -**General Availability (GA)** during 2018. Some sub-features, like self-hosting -or the configuration file API are still under active development. The -implementation of creating the cluster may change slightly as the tool evolves, -but the overall implementation should be pretty stable. Any commands under -`kubeadm alpha` are by definition, supported on an alpha level. +kubeadm's overall feature state is **GA**. Some sub-features, like the configuration +file API are still under active development. The implementation of creating the cluster +may change slightly as the tool evolves, but the overall implementation should be pretty stable. +Any commands under `kubeadm alpha` are by definition, supported on an alpha level. ### Support timeframes @@ -70,6 +69,7 @@ timeframe; which also applies to `kubeadm`. | v1.10.x | March 2018 | December 2018   | | v1.11.x | June 2018 | March 2019   | | v1.12.x | September 2018 | June 2019   | +| v1.13.x | December 2018 | September 2019   | {{% /capture %}} @@ -391,7 +391,7 @@ And once the CoreDNS pod is up and running, you can continue by joining your nod If your network is not working or CoreDNS is not in the Running state, check out our [troubleshooting docs](/docs/setup/independent/troubleshooting-kubeadm/). -### Master Isolation +### Control plane node isolation By default, your cluster will not schedule pods on the master for security reasons. If you want to be able to schedule pods on the master, e.g. for a @@ -508,7 +508,7 @@ and `scp` using that other user instead. The `admin.conf` file gives the user _superuser_ privileges over the cluster. This file should be used sparingly. For normal users, it's recommended to generate an unique credential to which you whitelist privileges. You can do -this with the `kubeadm alpha phase kubeconfig user --client-name ` +this with the `kubeadm alpha kubeconfig user --client-name ` command. That command will print out a KubeConfig file to STDOUT which you should save to a file and distribute to your user. After that, whitelist privileges by using `kubectl create (cluster)rolebinding`. @@ -598,6 +598,8 @@ kubeadm deb/rpm packages and binaries are built for amd64, arm (32-bit), arm64, following the [multi-platform proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multi-platform.md). +Multiplatform container images for the control plane and addons are also supported since v1.12. + Only some of the network providers offer solutions for all platforms. Please consult the list of network providers above or the documentation from each provider to figure out whether the provider supports your chosen platform. diff --git a/content/en/docs/setup/independent/ha-topology.md b/content/en/docs/setup/independent/ha-topology.md new file mode 100644 index 0000000000..2cd28f32ec --- /dev/null +++ b/content/en/docs/setup/independent/ha-topology.md @@ -0,0 +1,71 @@ +--- +reviewers: +- sig-cluster-lifecycle +title: Options for Highly Available Topology +content_template: templates/concept +weight: 50 +--- + +{{% capture overview %}} + +This page explains the two options for configuring the topology of your highly available (HA) Kubernetes clusters. + +You can set up an HA cluster: + +- With stacked control plane nodes, where etcd nodes are colocated with control plane nodes +- With external etcd nodes, where etcd runs on separate nodes from the control plane + +You should carefully consider the advantages and disadvantages of each topology before setting up an HA cluster. + +{{% /capture %}} + +{{% capture body %}} + +## Stacked etcd topology + +A stacked HA cluster is a [topology](https://en.wikipedia.org/wiki/Network_topology) where the distributed +data storage cluster provided by etcd is stacked on top of the cluster formed by the nodes managed by +kubeadm that run control plane components. + +Each control plane node runs an instance of the `kube-apiserver`, `kube-scheduler`, and `kube-controller-manager`. +The `kube-apiserver` is exposed to worker nodes using a load balancer. + +Each control plane node creates a local etcd member and this etcd member communicate only with +the `kube-apiserver` of this node. The same applies to the local `kube-controller-manager` +and `kube-scheduler` instances. + +This topology couples the control planes and etcd members on the same nodes. It is simpler to set up than a cluster +with external etcd nodes, and simpler to manage for replication. + +However, a stacked cluster runs the risk of failed coupling. If one node goes down, both an etcd member and a control +plane instance are lost, and redundancy is compromised. You can mitigate this risk by adding more control plane nodes. + +You should therefore run a minimum of three stacked control plane nodes for an HA cluster. + +This is the default topology in kubeadm. A local etcd member is created automatically +on control plane nodes when using `kubeadm init` and `kubeadm join --experimental-control-plane`. + +![Stacked etcd topology](/images/kubeadm/kubeadm-ha-topology-stacked-etcd.svg) + +## External etcd topology + +An HA cluster with external etcd is a [topology](https://en.wikipedia.org/wiki/Network_topology) where the distributed data storage cluster provided by etcd is external to the cluster formed by the nodes that run control plane components. + +Like the stacked etcd topology, each control plane node in an external etcd topology runs an instance of the `kube-apiserver`, `kube-scheduler`, and `kube-controller-manager`. And the `kube-apiserver` is exposed to worker nodes using a load balancer. However, etcd members run on separate hosts, and each etcd host communicates with the `kube-apiserver` of each control plane node. + +This topology decouples the control plane and etcd member. It therefore provides an HA setup where +losing a control plane instance or an etcd member has less impact and does not affect +the cluster redundancy as much as the stacked HA topology. + +However, this topology requires twice the number of hosts as the stacked HA topology. +A minimum of three hosts for control plane nodes and three hosts for etcd nodes are required for an HA cluster with this topology. + +![External etcd topology](/images/kubeadm/kubeadm-ha-topology-external-etcd.svg) + +{{% /capture %}} + +{{% capture whatsnext %}} + +- [Set up a highly available cluster with kubeadm](/docs/setup/independent/high-availability/) + +{{% /capture %}} \ No newline at end of file diff --git a/content/en/docs/setup/independent/high-availability.md b/content/en/docs/setup/independent/high-availability.md index e2b6f7b5dc..b43bb3377b 100644 --- a/content/en/docs/setup/independent/high-availability.md +++ b/content/en/docs/setup/independent/high-availability.md @@ -3,7 +3,7 @@ reviewers: - sig-cluster-lifecycle title: Creating Highly Available Clusters with kubeadm content_template: templates/task -weight: 50 +weight: 60 --- {{% capture overview %}} @@ -11,15 +11,23 @@ weight: 50 This page explains two different approaches to setting up a highly available Kubernetes cluster using kubeadm: -- With stacked masters. This approach requires less infrastructure. etcd members +- With stacked control plane nodes. This approach requires less infrastructure. The etcd members and control plane nodes are co-located. - With an external etcd cluster. This approach requires more infrastructure. The control plane nodes and etcd members are separated. +Before proceeding, you should carefully consideer which approach best meets the needs of your applications +and environment. [This comparison topic](/docs/setup/independent/ha-topology/) outlines the advantages and disadvantages of each. + Your clusters must run Kubernetes version 1.12 or later. You should also be aware that -setting up HA clusters with kubeadm is still experimental. You might encounter issues -with upgrading your clusters, for example. We encourage you to try either approach, -and provide feedback. +setting up HA clusters with kubeadm is still experimental and will be further simplified +in future versions. You might encounter issues with upgrading your clusters, for example. +We encourage you to try either approach, and provide us with feedback in the kubeadm +[issue tracker](https://github.com/kubernetes/kubeadm/issues/new). + +Note that the alpha feature gate `HighAvailability` is deprecated in v1.12 and removed in v1.13. + +See also [The HA upgrade documentation](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha). {{< caution >}} This page does not address running your cluster on a cloud provider. In a cloud @@ -40,9 +48,10 @@ For both methods you need this infrastructure: requirements](/docs/setup/independent/install-kubeadm/#before-you-begin) for the workers - Full network connectivity between all machines in the cluster (public or - private network is fine) -- SSH access from one device to all nodes in the system + private network) - sudo privileges on all machines +- SSH access from one device to all nodes in the system +- `kubeadm` and `kubelet` installed on all machines. `kubectl` is optional. For the external etcd cluster only, you also need: @@ -60,15 +69,60 @@ networking provider, make sure to replace any default values as needed. ## First steps for both methods {{< note >}} -All commands in this guide on any control plane or etcd node should be run as root. +**Note**: All commands on any control plane or etcd node should be +run as root. {{< /note >}} -- Find your pod CIDR. For details, see [the CNI network - documentation](/docs/setup/independent/create-cluster-kubeadm/#pod-network). - The example uses Calico, so the pod CIDR is `192.168.0.0/16`. +- Some CNI network plugins like Calico require a CIDR such as `192.168.0.0/16` and + some like Weave do not. See the see [the CNI network + documentation](/docs/setup/independent/create-cluster-kubeadm/#pod-network). + To add a pod CIDR set the `podSubnet: 192.168.0.0/16` field under + the `networking` object of `ClusterConfiguration`. + +### Create load balancer for kube-apiserver + +{{< note >}} +There are many configurations for load balancers. The following example is only one +option. Your cluster requirements may need a different configuration. +{{< /note >}} + +1. Create a kube-apiserver load balancer with a name that resolves to DNS. + + - In a cloud environment you should place your control plane nodes behind a TCP + forwarding load balancer. This load balancer distributes traffic to all + healthy control plane nodes in its target list. The health check for + an apiserver is a TCP check on the port the kube-apiserver listens on + (default value `:6443`). + + - It is not recommended to use an IP address directly in a cloud environment. + + - The load balancer must be able to communicate with all control plane nodes + on the apiserver port. It must also allow incoming traffic on its + listening port. + + - [HAProxy](http://www.haproxy.org/) can be used as a load balancer. + + - Make sure the address of the load balancer always matches + the address of kubeadm's `ControlPlaneEndpoint`. + +1. Add the first control plane nodes to the load balancer and test the + connection: + + ```sh + nc -v LOAD_BALANCER_IP PORT + ``` + + - A connection refused error is expected because the apiserver is not yet + running. A timeout, however, means the load balancer cannot communicate + with the control plane node. If a timeout occurs, reconfigure the load + balancer to communicate with the control plane node. + +1. Add the remaining control plane nodes to the load balancer target group. ### Configure SSH +SSH is required if you want to control all nodes from a single machine. + 1. Enable ssh-agent on your main device that has access to all other nodes in the system: @@ -97,167 +151,81 @@ All commands in this guide on any control plane or etcd node should be run as ro sudo -E -s ``` -### Create load balancer for kube-apiserver +## Stacked control plane and etcd nodes -{{< note >}} -There are many configurations for load balancers. The following example is only one -option. Your cluster requirements may need a different configuration. -{{< /note >}} +### Steps for the first control plane node -1. Create a kube-apiserver load balancer with a name that resolves to DNS. +1. On the first control plane node, create a configuration file called `kubeadm-config.yaml`: - - In a cloud environment you should place your control plane nodes behind a TCP - forwarding load balancer. This load balancer distributes traffic to all - healthy control plane nodes in its target list. The health check for - an apiserver is a TCP check on the port the kube-apiserver listens on - (default value `:6443`). + apiVersion: kubeadm.k8s.io/v1beta1 + kind: ClusterConfiguration + kubernetesVersion: stable + apiServer: + certSANs: + - "LOAD_BALANCER_DNS" + controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" - - It is not recommended to use an IP address directly in a cloud environment. + - `kubernetesVersion` should be set to the Kubernetes version to use. This + example uses `stable`. + - `controlPlaneEndpoint` should match the address or DNS and port of the load balancer. + - It's recommended that the versions of kubeadm, kubelet, kubectl and Kubernetes match. - - The load balancer must be able to communicate with all control plane nodes - on the apiserver port. It must also allow incoming traffic on its - listening port. - -1. Add the first control plane nodes to the load balancer and test the - connection: +1. Make sure that the node is in a clean state: ```sh - nc -v LOAD_BALANCER_IP PORT + sudo kubeadm init --config=kubeadm-config.yaml + ``` + + You should see something like: + + ```sh + ... + You can now join any number of machines by running the following on each node + as root: + + kubeadm join 192.168.0.200:6443 --token j04n3m.octy8zely83cy2ts --discovery-token-ca-cert-hash sha256:84938d2a22203a8e56a787ec0c6ddad7bc7dbd52ebabc62fd5f4dbea72b14d1f ``` - - A connection refused error is expected because the apiserver is not yet - running. A timeout, however, means the load balancer cannot communicate - with the control plane node. If a timeout occurs, reconfigure the load - balancer to communicate with the control plane node. +1. Copy this output to a text file. You will need it later to join other control plane nodes to the + cluster. -1. Add the remaining control plane nodes to the load balancer target group. +1. Apply the Weave CNI plugin: -## Stacked control plane nodes + ```sh + kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')" + ``` -### Bootstrap the first stacked control plane node +1. Type the following and watch the pods of the components get started: -{{< note >}} -Optionally replace the string `stable` with a different version of Kubernetes, for example `v1.12.0`. -{{< /note >}} + ```sh + kubectl get pod -n kube-system -w + ``` -1. Create a `kubeadm-config.yaml` template file: + - It's recommended that you join new control plane nodes only after the first node has finished initializing. - apiVersion: kubeadm.k8s.io/v1alpha3 - kind: ClusterConfiguration - kubernetesVersion: stable - apiServerCertSANs: - - "LOAD_BALANCER_DNS" - controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" - etcd: - local: - extraArgs: - name: "CP0_HOSTNAME" - listen-client-urls: "https://127.0.0.1:2379,https://CP0_IP:2379" - advertise-client-urls: "https://CP0_IP:2379" - listen-peer-urls: "https://CP0_IP:2380" - initial-advertise-peer-urls: "https://CP0_IP:2380" - initial-cluster: "CP0_HOSTNAME=https://CP0_IP:2380" - serverCertSANs: - - CP0_HOSTNAME - - CP0_IP - peerCertSANs: - - CP0_HOSTNAME - - CP0_IP - networking: - # This CIDR is a Calico default. Substitute or remove for your CNI provider. - podSubnet: "192.168.0.0/16" +1. Copy the certificate files from the first control plane node to the rest: -1. Replace the following variables in the template with the appropriate - values for your cluster: + In the following example, replace `CONTROL_PLANE_IPS` with the IP addresses of the + other control plane nodes. + ```sh + USER=ubuntu # customizable + CONTROL_PLANE_IPS="10.0.0.7 10.0.0.8" + for host in ${CONTROL_PLANE_IPS}; do + scp /etc/kubernetes/pki/ca.crt "${USER}"@$host: + scp /etc/kubernetes/pki/ca.key "${USER}"@$host: + scp /etc/kubernetes/pki/sa.key "${USER}"@$host: + scp /etc/kubernetes/pki/sa.pub "${USER}"@$host: + scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host: + scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host: + scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host:etcd-ca.crt + scp /etc/kubernetes/pki/etcd/ca.key "${USER}"@$host:etcd-ca.key + scp /etc/kubernetes/admin.conf "${USER}"@$host: + done + ``` - * `LOAD_BALANCER_DNS` - * `LOAD_BALANCER_PORT` - * `CP0_HOSTNAME` - * `CP0_IP` +### Steps for the rest of the control plane nodes -1. Run `kubeadm init --config kubeadm-config.yaml` - -### Copy required files to other control plane nodes - -The following certificates and other required files were created when you ran `kubeadm init`. -Copy these files to your other control plane nodes: - -- `/etc/kubernetes/pki/ca.crt` -- `/etc/kubernetes/pki/ca.key` -- `/etc/kubernetes/pki/sa.key` -- `/etc/kubernetes/pki/sa.pub` -- `/etc/kubernetes/pki/front-proxy-ca.crt` -- `/etc/kubernetes/pki/front-proxy-ca.key` -- `/etc/kubernetes/pki/etcd/ca.crt` -- `/etc/kubernetes/pki/etcd/ca.key` - -Copy the admin kubeconfig to the other control plane nodes: - -- `/etc/kubernetes/admin.conf` - -In the following example, replace -`CONTROL_PLANE_IPS` with the IP addresses of the other control plane nodes. - -```sh -USER=ubuntu # customizable -CONTROL_PLANE_IPS="10.0.0.7 10.0.0.8" -for host in ${CONTROL_PLANE_IPS}; do - scp /etc/kubernetes/pki/ca.crt "${USER}"@$host: - scp /etc/kubernetes/pki/ca.key "${USER}"@$host: - scp /etc/kubernetes/pki/sa.key "${USER}"@$host: - scp /etc/kubernetes/pki/sa.pub "${USER}"@$host: - scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host: - scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host: - scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host:etcd-ca.crt - scp /etc/kubernetes/pki/etcd/ca.key "${USER}"@$host:etcd-ca.key - scp /etc/kubernetes/admin.conf "${USER}"@$host: -done -``` - -{{< note >}} -Remember that your config may differ from this example. -{{< /note >}} - -### Add the second stacked control plane node - -1. Create a second, different `kubeadm-config.yaml` template file: - - apiVersion: kubeadm.k8s.io/v1alpha3 - kind: ClusterConfiguration - kubernetesVersion: stable - apiServerCertSANs: - - "LOAD_BALANCER_DNS" - controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" - etcd: - local: - extraArgs: - name: "CP1_HOSTNAME" - listen-client-urls: "https://127.0.0.1:2379,https://CP1_IP:2379" - advertise-client-urls: "https://CP1_IP:2379" - listen-peer-urls: "https://CP1_IP:2380" - initial-advertise-peer-urls: "https://CP1_IP:2380" - initial-cluster: "CP0_HOSTNAME=https://CP0_IP:2380,CP1_HOSTNAME=https://CP1_IP:2380" - initial-cluster-state: existing - serverCertSANs: - - CP1_HOSTNAME - - CP1_IP - peerCertSANs: - - CP1_HOSTNAME - - CP1_IP - networking: - # This CIDR is a calico default. Substitute or remove for your CNI provider. - podSubnet: "192.168.0.0/16" - -1. Replace the following variables in the template with the appropriate values for your cluster: - - - `LOAD_BALANCER_DNS` - - `LOAD_BALANCER_PORT` - - `CP0_HOSTNAME` - - `CP0_IP` - - `CP1_HOSTNAME` - - `CP1_IP` - -1. Move the copied files to the correct locations: +1. Move the files created by the previous step where `scp` was used: ```sh USER=ubuntu # customizable @@ -273,180 +241,54 @@ Remember that your config may differ from this example. mv /home/${USER}/admin.conf /etc/kubernetes/admin.conf ``` -1. Run the kubeadm phase commands to bootstrap the kubelet: + This process writes all the requested files in the `/etc/kubernetes` folder. + +1. Start `kubeadm join` on this node using the join command that was previously given to you by `kubeadm init` on + the first node. It should look something like this: ```sh - kubeadm alpha phase certs all --config kubeadm-config.yaml - kubeadm alpha phase kubelet config write-to-disk --config kubeadm-config.yaml - kubeadm alpha phase kubelet write-env-file --config kubeadm-config.yaml - kubeadm alpha phase kubeconfig kubelet --config kubeadm-config.yaml - systemctl start kubelet + sudo kubeadm join 192.168.0.200:6443 --token j04n3m.octy8zely83cy2ts --discovery-token-ca-cert-hash sha256:84938d2a22203a8e56a787ec0c6ddad7bc7dbd52ebabc62fd5f4dbea72b14d1f --experimental-control-plane ``` -1. Run the commands to add the node to the etcd cluster: + - Notice the addition of the `--experimental-control-plane` flag. This flag automates joining this + control plane node to the cluster. + +1. Type the following and watch the pods of the components get started: ```sh - export CP0_IP=10.0.0.7 - export CP0_HOSTNAME=cp0 - export CP1_IP=10.0.0.8 - export CP1_HOSTNAME=cp1 - - kubeadm alpha phase etcd local --config kubeadm-config.yaml - export KUBECONFIG=/etc/kubernetes/admin.conf - kubectl exec -n kube-system etcd-${CP0_HOSTNAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${CP0_IP}:2379 member add ${CP1_HOSTNAME} https://${CP1_IP}:2380 + kubectl get pod -n kube-system -w ``` - - This command causes the etcd cluster to become unavailable for a - brief period, after the node is added to the running cluster, and before the - new node is joined to the etcd cluster. +1. Repeat these steps for the rest of the control plane nodes. -1. Deploy the control plane components and mark the node as a master: +## External etcd nodes - ```sh - kubeadm alpha phase kubeconfig all --config kubeadm-config.yaml - kubeadm alpha phase controlplane all --config kubeadm-config.yaml - kubeadm alpha phase kubelet config annotate-cri --config kubeadm-config.yaml - kubeadm alpha phase mark-master --config kubeadm-config.yaml - ``` - -### Add the third stacked control plane node - -1. Create a third, different `kubeadm-config.yaml` template file: - - apiVersion: kubeadm.k8s.io/v1alpha3 - kind: ClusterConfiguration - kubernetesVersion: stable - apiServerCertSANs: - - "LOAD_BALANCER_DNS" - controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" - etcd: - local: - extraArgs: - name: "CP2_HOSTNAME" - listen-client-urls: "https://127.0.0.1:2379,https://CP2_IP:2379" - advertise-client-urls: "https://CP2_IP:2379" - listen-peer-urls: "https://CP2_IP:2380" - initial-advertise-peer-urls: "https://CP2_IP:2380" - initial-cluster: "CP0_HOSTNAME=https://CP0_IP:2380,CP1_HOSTNAME=https://CP1_IP:2380,CP2_HOSTNAME=https://CP2_IP:2380" - initial-cluster-state: existing - serverCertSANs: - - CP2_HOSTNAME - - CP2_IP - peerCertSANs: - - CP2_HOSTNAME - - CP2_IP - networking: - # This CIDR is a calico default. Substitute or remove for your CNI provider. - podSubnet: "192.168.0.0/16" - -1. Replace the following variables in the template with the appropriate values for your cluster: - - - `LOAD_BALANCER_DNS` - - `LOAD_BALANCER_PORT` - - `CP0_HOSTNAME` - - `CP0_IP` - - `CP1_HOSTNAME` - - `CP1_IP` - - `CP2_HOSTNAME` - - `CP2_IP` - -1. Move the copied files to the correct locations: - - ```sh - USER=ubuntu # customizable - mkdir -p /etc/kubernetes/pki/etcd - mv /home/${USER}/ca.crt /etc/kubernetes/pki/ - mv /home/${USER}/ca.key /etc/kubernetes/pki/ - mv /home/${USER}/sa.pub /etc/kubernetes/pki/ - mv /home/${USER}/sa.key /etc/kubernetes/pki/ - mv /home/${USER}/front-proxy-ca.crt /etc/kubernetes/pki/ - mv /home/${USER}/front-proxy-ca.key /etc/kubernetes/pki/ - mv /home/${USER}/etcd-ca.crt /etc/kubernetes/pki/etcd/ca.crt - mv /home/${USER}/etcd-ca.key /etc/kubernetes/pki/etcd/ca.key - mv /home/${USER}/admin.conf /etc/kubernetes/admin.conf - ``` - -1. Run the kubeadm phase commands to bootstrap the kubelet: - - ```sh - kubeadm alpha phase certs all --config kubeadm-config.yaml - kubeadm alpha phase kubelet config write-to-disk --config kubeadm-config.yaml - kubeadm alpha phase kubelet write-env-file --config kubeadm-config.yaml - kubeadm alpha phase kubeconfig kubelet --config kubeadm-config.yaml - systemctl start kubelet - ``` - -1. Run the commands to add the node to the etcd cluster: - - ```sh - export CP0_IP=10.0.0.7 - export CP0_HOSTNAME=cp0 - export CP2_IP=10.0.0.9 - export CP2_HOSTNAME=cp2 - - export KUBECONFIG=/etc/kubernetes/admin.conf - kubectl exec -n kube-system etcd-${CP0_HOSTNAME} -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${CP0_IP}:2379 member add ${CP2_HOSTNAME} https://${CP2_IP}:2380 - kubeadm alpha phase etcd local --config kubeadm-config.yaml - ``` - -1. Deploy the control plane components and mark the node as a master: - - ```sh - kubeadm alpha phase kubeconfig all --config kubeadm-config.yaml - kubeadm alpha phase controlplane all --config kubeadm-config.yaml - kubeadm alpha phase kubelet config annotate-cri --config kubeadm-config.yaml - kubeadm alpha phase mark-master --config kubeadm-config.yaml - ``` - -## External etcd - -### Set up the cluster +### Set up the etcd cluster - Follow [these instructions](/docs/setup/independent/setup-ha-etcd-with-kubeadm/) - to set up the etcd cluster. - -#### Copy required files from an etcd node to all control plane nodes - -In the following example, replace `USER` and `CONTROL_PLANE_HOSTS` values with values -for your environment. - -```sh -# Make a list of required etcd certificate files -cat << EOF > etcd-pki-files.txt -/etc/kubernetes/pki/etcd/ca.crt -/etc/kubernetes/pki/apiserver-etcd-client.crt -/etc/kubernetes/pki/apiserver-etcd-client.key -EOF - -# create the archive -tar -czf etcd-pki.tar.gz -T etcd-pki-files.txt - -# copy the archive to the control plane nodes -USER=ubuntu -CONTROL_PLANE_HOSTS="10.0.0.7 10.0.0.8 10.0.0.9" -for host in $CONTROL_PLANE_HOSTS; do - scp etcd-pki.tar.gz "${USER}"@$host: -done -``` + to set up the etcd cluster. ### Set up the first control plane node -{{< note >}} -Optionally replace the string `stable` with a different version of Kubernetes, for example `v1.11.3`. -{{< /note >}} +1. Copy the following files from any node from the etcd cluster to this node: -1. Extract the etcd certificates + ```sh + export CONTROL_PLANE="ubuntu@10.0.0.7" + +scp /etc/kubernetes/pki/etcd/ca.crt "${CONTROL_PLANE}": + +scp /etc/kubernetes/pki/apiserver-etcd-client.crt "${CONTROL_PLANE}": + +scp /etc/kubernetes/pki/apiserver-etcd-client.key "${CONTROL_PLANE}": + ``` - mkdir -p /etc/kubernetes/pki - tar -xzf etcd-pki.tar.gz -C /etc/kubernetes/pki --strip-components=3 + - Replace the value of `CONTROL_PLANE` with the `user@host` of this machine. -1. Create a `kubeadm-config.yaml`: +1. Create a file called `kubeadm-config.yaml` with the following contents: - apiVersion: kubeadm.k8s.io/v1alpha3 + apiVersion: kubeadm.k8s.io/v1beta1 kind: ClusterConfiguration kubernetesVersion: stable - apiServerCertSANs: - - "LOAD_BALANCER_DNS" + apiServer: + certSANs: + - "LOAD_BALANCER_DNS" controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" etcd: external: @@ -457,82 +299,38 @@ Optionally replace the string `stable` with a different version of Kubernetes, f caFile: /etc/kubernetes/pki/etcd/ca.crt certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key - networking: - # This CIDR is a calico default. Substitute or remove for your CNI provider. - podSubnet: "192.168.0.0/16" -1. Replace the following variables in the template with the appropriate values for your cluster: + - The difference between stacked etcd and external etcd here is that we are using the `external` field for `etcd` in the kubeadm config. In the case of the stacked etcd topology this is managed automatically. - - `LOAD_BALANCER_DNS` - - `LOAD_BALANCER_PORT` - - `ETCD_0_IP` - - `ETCD_1_IP` - - `ETCD_2_IP` + - Replace the following variables in the template with the appropriate values for your cluster: -1. Run `kubeadm init --config kubeadm-config.yaml` -1. Copy the output from the join command + - `LOAD_BALANCER_DNS` + - `LOAD_BALANCER_PORT` + - `ETCD_0_IP` + - `ETCD_1_IP` + - `ETCD_2_IP` -### Copy required files to the correct locations +1. Run `kubeadm init --config kubeadm-config.yaml` on this node. -The following pki files were created during the `kubeadm init` step and must be shared with -all other control plane nodes. +1. Write the join command that is returned to a text file for later use. -- `/etc/kubernetes/pki/ca.crt` -- `/etc/kubernetes/pki/ca.key` -- `/etc/kubernetes/pki/sa.key` -- `/etc/kubernetes/pki/sa.pub` -- `/etc/kubernetes/pki/front-proxy-ca.crt` -- `/etc/kubernetes/pki/front-proxy-ca.key` +1. Apply the Weave CNI plugin: -In the following example, replace the list of -`CONTROL_PLANE_IPS` values with the IP addresses of the other control plane nodes. + ```sh + kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')" + ``` -```sh -# make a list of required kubernetes certificate files -cat << EOF > certificate_files.txt -/etc/kubernetes/pki/ca.crt -/etc/kubernetes/pki/ca.key -/etc/kubernetes/pki/sa.key -/etc/kubernetes/pki/sa.pub -/etc/kubernetes/pki/front-proxy-ca.crt -/etc/kubernetes/pki/front-proxy-ca.key -EOF +### Steps for the rest of the control plane nodes -# create the archive -tar -czf control-plane-certificates.tar.gz -T certificate_files.txt +To add the rest of the control plane nodes, follow [these instructions](#steps-for-the-rest-of-the-control-plane-nodes). +The steps are the same as for the stacked etcd setup, with the exception that a local +etcd member is not created. -USER=ubuntu # customizable -CONTROL_PLANE_IPS="10.0.0.7 10.0.0.8" -for host in ${CONTROL_PLANE_IPS}; do - scp control-plane-certificates.tar.gz "${USER}"@$host: -done -``` +To summarize: -### Set up the other control plane nodes - -1. Extract the required certificates - - mkdir -p /etc/kubernetes/pki - tar -xzf etcd-pki.tar.gz -C /etc/kubernetes/pki --strip-components 3 - tar -xzf control-plane-certificates.tar.gz -C /etc/kubernetes/pki --strip-components 3 - -1. Verify the location of the copied files. - Your `/etc/kubernetes` directory should look like this: - - - `/etc/kubernetes/pki/apiserver-etcd-client.crt` - - `/etc/kubernetes/pki/apiserver-etcd-client.key` - - `/etc/kubernetes/pki/ca.crt` - - `/etc/kubernetes/pki/ca.key` - - `/etc/kubernetes/pki/front-proxy-ca.crt` - - `/etc/kubernetes/pki/front-proxy-ca.key` - - `/etc/kubernetes/pki/sa.key` - - `/etc/kubernetes/pki/sa.pub` - - `/etc/kubernetes/pki/etcd/ca.crt` - -1. Run the copied `kubeadm join` command from above. Add the flag "--experimental-control-plane". - The final command will look something like this: - - kubeadm join ha.k8s.example.com:6443 --token 5ynki1.3erp9i3yo7gqg1nv --discovery-token-ca-cert-hash sha256:a00055bd8c710a9906a3d91b87ea02976334e1247936ac061d867a0f014ecd81 --experimental-control-plane +- Make sure the first control plane node is fully initialized. +- Copy certificates between the first control plane node and the other control plane nodes. +- Join each control plane node with the join command you saved to a text file, plus add the `--experimental-control-plane` flag. ## Common tasks after bootstrapping control plane @@ -545,6 +343,6 @@ in the master configuration file. ### Install workers Each worker node can now be joined to the cluster with the command returned from any of the -`kubeadm init` commands. +`kubeadm init` commands. The flag `--experimental-control-plane` should not be added to worker nodes. {{% /capture %}} diff --git a/content/en/docs/setup/independent/kubelet-integration.md b/content/en/docs/setup/independent/kubelet-integration.md index 9b55041733..6825c135af 100644 --- a/content/en/docs/setup/independent/kubelet-integration.md +++ b/content/en/docs/setup/independent/kubelet-integration.md @@ -3,7 +3,7 @@ reviewers: - sig-cluster-lifecycle title: Configuring each kubelet in your cluster using kubeadm content_template: templates/concept -weight: 40 +weight: 80 --- {{% capture overview %}} diff --git a/content/en/docs/setup/independent/setup-ha-etcd-with-kubeadm.md b/content/en/docs/setup/independent/setup-ha-etcd-with-kubeadm.md index 2d5ea7c059..1567bf0dda 100644 --- a/content/en/docs/setup/independent/setup-ha-etcd-with-kubeadm.md +++ b/content/en/docs/setup/independent/setup-ha-etcd-with-kubeadm.md @@ -3,7 +3,7 @@ reviewers: - sig-cluster-lifecycle title: Set up a High Availability etcd cluster with kubeadm content_template: templates/task -weight: 60 +weight: 70 --- {{% capture overview %}} @@ -83,7 +83,7 @@ this example. HOST=${ETCDHOSTS[$i]} NAME=${NAMES[$i]} cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml - apiVersion: "kubeadm.k8s.io/v1alpha3" + apiVersion: "kubeadm.k8s.io/v1beta1" kind: ClusterConfiguration etcd: local: @@ -114,7 +114,7 @@ this example. generated the configuration files for kubeadm). ``` - kubeadm alpha phase certs etcd-ca + kubeadm init phase certs etcd-ca ``` This creates two files @@ -125,25 +125,25 @@ this example. 1. Create certificates for each member ```sh - kubeadm alpha phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml - kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml - kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml - kubeadm alpha phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml + kubeadm init phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml + kubeadm init phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml + kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml + kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml cp -R /etc/kubernetes/pki /tmp/${HOST2}/ # cleanup non-reusable certificates find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete - kubeadm alpha phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml - kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml - kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml - kubeadm alpha phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml + kubeadm init phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml + kubeadm init phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml + kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml + kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml cp -R /etc/kubernetes/pki /tmp/${HOST1}/ find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete - kubeadm alpha phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml - kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml - kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml - kubeadm alpha phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml + kubeadm init phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml + kubeadm init phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml + kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml + kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml # No need to move the certs because they are for HOST0 # clean up certs that should not be copied off this host @@ -233,9 +233,9 @@ this example. for etcd. ```sh - root@HOST0 $ kubeadm alpha phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml - root@HOST1 $ kubeadm alpha phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml - root@HOST2 $ kubeadm alpha phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml + root@HOST0 $ kubeadm init phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml + root@HOST1 $ kubeadm init phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml + root@HOST2 $ kubeadm init phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml ``` 1. Optional: Check the cluster health @@ -243,7 +243,7 @@ this example. ```sh docker run --rm -it \ --net host \ - -v /etc/kubernetes:/etc/kubernetes quay.io/coreos/etcd:v3.2.18 etcdctl \ + -v /etc/kubernetes:/etc/kubernetes quay.io/coreos/etcd:${ETCD_TAG} etcdctl \ --cert-file /etc/kubernetes/pki/etcd/peer.crt \ --key-file /etc/kubernetes/pki/etcd/peer.key \ --ca-file /etc/kubernetes/pki/etcd/ca.crt \ @@ -251,6 +251,8 @@ this example. ... cluster is healthy ``` + - Set `${ETCD_TAG}` to the version tag of your etcd image. For example `v3.2.24`. + - Set `${HOST0}`to the IP address of the host you are testing. {{% /capture %}} diff --git a/content/en/docs/setup/independent/troubleshooting-kubeadm.md b/content/en/docs/setup/independent/troubleshooting-kubeadm.md index a5ab81a62c..b3a9e6fba3 100644 --- a/content/en/docs/setup/independent/troubleshooting-kubeadm.md +++ b/content/en/docs/setup/independent/troubleshooting-kubeadm.md @@ -1,7 +1,7 @@ --- title: Troubleshooting kubeadm content_template: templates/concept -weight: 70 +weight: 90 --- {{% capture overview %}} @@ -194,7 +194,7 @@ Error from server: Get https://10.19.0.41:10250/containerLogs/default/mysql-ddc6 curl http://169.254.169.254/metadata/v1/interfaces/public/0/anchor_ipv4/address ``` - The workaround is to tell `kubelet` which IP to use using `--node-ip`. When using Digital Ocean, it can be the public one (assigned to `eth0`) or the private one (assigned to `eth1`) should you want to use the optional private network. The [`KubeletExtraArgs` section of the kubeadm `NodeRegistrationOptions` structure](https://github.com/kubernetes/kubernetes/blob/release-1.12/cmd/kubeadm/app/apis/kubeadm/v1alpha3/types.go#L163-L166) can be used for this. + The workaround is to tell `kubelet` which IP to use using `--node-ip`. When using Digital Ocean, it can be the public one (assigned to `eth0`) or the private one (assigned to `eth1`) should you want to use the optional private network. The [`KubeletExtraArgs` section of the kubeadm `NodeRegistrationOptions` structure](https://github.com/kubernetes/kubernetes/blob/release-1.13/cmd/kubeadm/app/apis/kubeadm/v1beta1/types.go) can be used for this. Then restart `kubelet`: @@ -203,46 +203,6 @@ Error from server: Get https://10.19.0.41:10250/containerLogs/default/mysql-ddc6 systemctl restart kubelet ``` -## Services with externalTrafficPolicy=Local are not reachable - -On nodes where the hostname for the kubelet is overridden using the `--hostname-override` option, kube-proxy will default to treating 127.0.0.1 as the node IP, which results in rejecting connections for Services configured for `externalTrafficPolicy=Local`. This situation can be verified by checking the output of `kubectl -n kube-system logs `: - -```sh -W0507 22:33:10.372369 1 server.go:586] Failed to retrieve node info: nodes "ip-10-0-23-78" not found -W0507 22:33:10.372474 1 proxier.go:463] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP -``` - -A workaround for this is to modify the kube-proxy DaemonSet in the following way: - -```sh -kubectl -n kube-system patch --type json daemonset kube-proxy -p "$(cat <<'EOF' -[ - { - "op": "add", - "path": "/spec/template/spec/containers/0/env", - "value": [ - { - "name": "NODE_NAME", - "valueFrom": { - "fieldRef": { - "apiVersion": "v1", - "fieldPath": "spec.nodeName" - } - } - } - ] - }, - { - "op": "add", - "path": "/spec/template/spec/containers/0/command/-", - "value": "--hostname-override=${NODE_NAME}" - } -] -EOF -)" - -``` - ## `coredns` pods have `CrashLoopBackOff` or `Error` state If you have nodes that are running SELinux with an older version of Docker you might experience a scenario diff --git a/content/en/docs/setup/release/notes.md b/content/en/docs/setup/release/notes.md index c79d745f1b..6663698f44 100644 --- a/content/en/docs/setup/release/notes.md +++ b/content/en/docs/setup/release/notes.md @@ -1,856 +1,1330 @@ --- -title: v1.12 Release Notes +title: v1.13 Release Notes --- + -- [v1.12.0-rc.2](#v1120-rc2) - - [Downloads for v1.12.0-rc.2](#downloads-for-v1120-rc2) +- [v1.13.0](#v1130) + - [Downloads for v1.13.0](#downloads-for-v1130) - [Client Binaries](#client-binaries) - [Server Binaries](#server-binaries) - [Node Binaries](#node-binaries) - - [Changelog since v1.12.0-rc.1](#changelog-since-v1120-rc1) - - [Other notable changes](#other-notable-changes) -- [v1.12.0-rc.1](#v1120-rc1) - - [Downloads for v1.12.0-rc.1](#downloads-for-v1120-rc1) +- [Kubernetes 1.13 Release Notes](#kubernetes-113-release-notes) + - [Security Content](#security-content) + - [Urgent Upgrade Notes](#urgent-upgrade-notes) + - [(No, really, you MUST do this before you upgrade)](#no-really-you-must-do-this-before-you-upgrade) + - [Known Issues](#known-issues) + - [Deprecations](#deprecations) + - [Major Themes](#major-themes) + - [SIG API Machinery](#sig-api-machinery) + - [SIG Auth](#sig-auth) + - [SIG AWS](#sig-aws) + - [SIG Azure](#sig-azure) + - [SIG Big Data](#sig-big-data) + - [SIG CLI](#sig-cli) + - [SIG Cloud Provider](#sig-cloud-provider) + - [SIG Cluster Lifecycle](#sig-cluster-lifecycle) + - [SIG IBM Cloud](#sig-ibm-cloud) + - [SIG Multicluster](#sig-multicluster) + - [SIG Network](#sig-network) + - [SIG Node](#sig-node) + - [SIG Openstack](#sig-openstack) + - [SIG Scalability](#sig-scalability) + - [SIG Scheduling](#sig-scheduling) + - [SIG Service Catalog](#sig-service-catalog) + - [SIG Storage](#sig-storage) + - [SIG UI](#sig-ui) + - [SIG VMWare](#sig-vmware) + - [SIG Windows](#sig-windows) + - [New Features](#new-features) + - [Release Notes From SIGs](#release-notes-from-sigs) + - [SIG API Machinery](#sig-api-machinery-1) + - [SIG Auth](#sig-auth-1) + - [SIG Autoscaling](#sig-autoscaling) + - [SIG AWS](#sig-aws-1) + - [SIG Azure](#sig-azure-1) + - [SIG CLI](#sig-cli-1) + - [SIG Cloud Provider](#sig-cloud-provider-1) + - [SIG Cluster Lifecycle](#sig-cluster-lifecycle-1) + - [SIG GCP](#sig-gcp) + - [SIG Network](#sig-network-1) + - [SIG Node](#sig-node-1) + - [SIG OpenStack](#sig-openstack-1) + - [SIG Release](#sig-release) + - [SIG Scheduling](#sig-scheduling-1) + - [SIG Storage](#sig-storage-1) + - [SIG Windows](#sig-windows-1) + - [External Dependencies](#external-dependencies) +- [v1.13.0-rc.2](#v1130-rc2) + - [Downloads for v1.13.0-rc.2](#downloads-for-v1130-rc2) - [Client Binaries](#client-binaries-1) - [Server Binaries](#server-binaries-1) - [Node Binaries](#node-binaries-1) - - [Changelog since v1.12.0-beta.2](#changelog-since-v1120-beta2) - - [Action Required](#action-required) - - [Other notable changes](#other-notable-changes-1) -- [v1.12.0-beta.2](#v1120-beta2) - - [Downloads for v1.12.0-beta.2](#downloads-for-v1120-beta2) + - [Changelog since v1.13.0-rc.1](#changelog-since-v1130-rc1) + - [Other notable changes](#other-notable-changes) +- [v1.13.0-rc.1](#v1130-rc1) + - [Downloads for v1.13.0-rc.1](#downloads-for-v1130-rc1) - [Client Binaries](#client-binaries-2) - [Server Binaries](#server-binaries-2) - [Node Binaries](#node-binaries-2) - - [Changelog since v1.12.0-beta.1](#changelog-since-v1120-beta1) - - [Action Required](#action-required-1) - - [Other notable changes](#other-notable-changes-2) -- [v1.12.0-beta.1](#v1120-beta1) - - [Downloads for v1.12.0-beta.1](#downloads-for-v1120-beta1) + - [Changelog since v1.13.0-beta.2](#changelog-since-v1130-beta2) + - [Other notable changes](#other-notable-changes-1) +- [v1.13.0-beta.2](#v1130-beta2) + - [Downloads for v1.13.0-beta.2](#downloads-for-v1130-beta2) - [Client Binaries](#client-binaries-3) - [Server Binaries](#server-binaries-3) - [Node Binaries](#node-binaries-3) - - [Changelog since v1.12.0-alpha.1](#changelog-since-v1120-alpha1) - - [Action Required](#action-required-2) - - [Other notable changes](#other-notable-changes-3) -- [v1.12.0-alpha.1](#v1120-alpha1) - - [Downloads for v1.12.0-alpha.1](#downloads-for-v1120-alpha1) + - [Changelog since v1.13.0-beta.1](#changelog-since-v1130-beta1) + - [Other notable changes](#other-notable-changes-2) +- [v1.13.0-beta.1](#v1130-beta1) + - [Downloads for v1.13.0-beta.1](#downloads-for-v1130-beta1) - [Client Binaries](#client-binaries-4) - [Server Binaries](#server-binaries-4) - [Node Binaries](#node-binaries-4) - - [Changelog since v1.11.0](#changelog-since-v1110) - - [Action Required](#action-required-3) + - [Changelog since v1.13.0-alpha.3](#changelog-since-v1130-alpha3) + - [Action Required](#action-required) + - [Other notable changes](#other-notable-changes-3) +- [v1.13.0-alpha.3](#v1130-alpha3) + - [Downloads for v1.13.0-alpha.3](#downloads-for-v1130-alpha3) + - [Client Binaries](#client-binaries-5) + - [Server Binaries](#server-binaries-5) + - [Node Binaries](#node-binaries-5) + - [Changelog since v1.13.0-alpha.2](#changelog-since-v1130-alpha2) - [Other notable changes](#other-notable-changes-4) +- [v1.13.0-alpha.2](#v1130-alpha2) + - [Downloads for v1.13.0-alpha.2](#downloads-for-v1130-alpha2) + - [Client Binaries](#client-binaries-6) + - [Server Binaries](#server-binaries-6) + - [Node Binaries](#node-binaries-6) + - [Changelog since v1.13.0-alpha.1](#changelog-since-v1130-alpha1) + - [Other notable changes](#other-notable-changes-5) +- [v1.13.0-alpha.1](#v1130-alpha1) + - [Downloads for v1.13.0-alpha.1](#downloads-for-v1130-alpha1) + - [Client Binaries](#client-binaries-7) + - [Server Binaries](#server-binaries-7) + - [Node Binaries](#node-binaries-7) + - [Changelog since v1.12.0](#changelog-since-v1120) + - [Action Required](#action-required-1) + - [Other notable changes](#other-notable-changes-6) -# v1.12.0-rc.2 +# v1.13.0 -[Documentation](https://docs.k8s.io) & [Examples](https://releases.k8s.io/release-1.12/examples) +[Documentation](https://docs.k8s.io) -## Downloads for v1.12.0-rc.2 +## Downloads for v1.13.0 -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes.tar.gz) | `184ea437bc72d0e6a4c96b964de53181273e919a1d4785515da3406c7e982bf5` -[kubernetes-src.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-src.tar.gz) | `aee82938827ef05ab0ee81bac42f4f79fff126294469868d02efb3426717d71e` +[kubernetes.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes.tar.gz) | `7b6a81c9f1b852b1e889c1b62281569a4b8853c79e5675b0910d941dfa7863c97f244f6d607aae3faf60bccd596dedb9d136b7fffeae199876e780904fd9f31e` +[kubernetes-src.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-src.tar.gz) | `844b9fbba21374dd190c8f12dd0e5b3303dd2cd7ad25f241d6f7e46f74adf6987afad021553521d4f479c19d87aa8d4d5be77ac7a6715d31a9187a5bab3b397b` ### Client Binaries -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-client-darwin-386.tar.gz) | `40ed3ef9bbc4fad7787dd14eae952edf06d40e1094604bc6d10209b8778c3121` -[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-client-darwin-amd64.tar.gz) | `a317fe3801ea5387ce474b9759a7e28ede8324587f79935a7a945da44c99a4b2` -[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-client-linux-386.tar.gz) | `cd61b4b71d6b739582c02b5be1d87d928507bc59f64ee72629a920cc529a0941` -[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-client-linux-amd64.tar.gz) | `306af04fc18ca2588e16fd831358df50a2cb02219687b543073836f835de8583` -[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-client-linux-arm.tar.gz) | `497584f2686339cce857cff1ebf4ed10dcd63f4684a03c242b0828fcd307be4c` -[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-client-linux-arm64.tar.gz) | `1dfbb8c299f5af15239ef39135a6c8a52ee4c234764ee0437d8f707e636c9124` -[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-client-linux-ppc64le.tar.gz) | `668d6f35c5f6adcd25584d9ef74c549db13ffca9d93b4bc8d25609a8e5837640` -[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-client-linux-s390x.tar.gz) | `8a8e205c38858bd9d161115e5e2870c6cfc9c82e189d156e7062e6fa979c3fda` -[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-client-windows-386.tar.gz) | `cdef48279c22cc8c764e43a4b9c2a86f02f21c80abbbcd48041fb1e89fb1eb67` -[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-client-windows-amd64.tar.gz) | `50621a3d2b1550c69325422c6dce78f5690574b35d3778dd3afcf698b57f0f54` +[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-client-darwin-386.tar.gz) | `0c010351acb660a75122feb876c9887d46ec2cb466872dd073b7f5b26fdadd96888a350e01606f2ae43606a5a4ab2d9309441f4357cee924b19688f9b02c55dc` +[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-client-darwin-amd64.tar.gz) | `c2c40bd202900124f4e9458b067a1e1fc040030dc84ce9bcc6a5beb263de05892c16f3bdafb8d854e343e71f086207f390fd0b60f6e32e770c73294b053da6e4` +[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-client-linux-386.tar.gz) | `5f5449be103b103d72a4e2b1028ab014cf7f74781166327f2ae284e4f5ecb539f6b60f36b8f7c7be0ae43dfb30661b2672dd93a1fa7e26d6c67498672674bf12` +[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-client-linux-amd64.tar.gz) | `61a6cd3b1fb34507e0b762a45da09d88e34921985970a2ba594e0e5af737d94c966434b4e9f8e84fb73a0aeb5fa3e557344cd2eb902bf73c67d4b4bff33c6831` +[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-client-linux-arm.tar.gz) | `dd5591e2b88c347759a138c4d2436a0f5252341d0e8c9fbab16b8f151e2744cbdd0c8583555a451425bc471f11b688ce568d9245caf8a278cbac2b343fdead89` +[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-client-linux-arm64.tar.gz) | `894ed30261598ebf3485f3575e95f85e3c353f4d834bf9a6ea53b265427704b43fba5403fbc4d522b3f02afb08e6afaae200af1fe57996291a7c74398ec2fe17` +[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-client-linux-ppc64le.tar.gz) | `6c26c807fc730ea736fda75dc57ac73395ba78bb828fffeee18b385be550d8f3ba2bbc27a52a8f15bcbbe68218c7945d9fb725e6759c117422bc0a632c110670` +[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-client-linux-s390x.tar.gz) | `41e6e972de77c0bde22fdd779ea64e731b60f32e97e78a024f33fc3e33a3b364b7f77ece7d3c64ad85b7f8fe7c8fc6d6892098a3362d1fe01ebf3d551fe2bf37` +[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-client-windows-386.tar.gz) | `442229e5030452901b924a94e7a879d4085597a4f201a5b3fc5ac9806cab5830c836cfa7a33e8f1693fe2e8badc4047bf227d7fb00c537fb1fb4cb7639de5455` +[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-client-windows-amd64.tar.gz) | `a11a8e8e732e7292781b9cb1de6e3e41683f95fb3fefc2b1a7b5fb1f064a0d80c0833876d931675135778457d81de9ed2e81caee4b3eb27d9f23c7b722b17442` ### Server Binaries -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-server-linux-amd64.tar.gz) | `87a8438887a2daa199508aae591b158025860b8381c64cbe9b1d0c06c4eebde9` -[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-server-linux-arm.tar.gz) | `f65be73870a0e564ef8ce1b6bb2b75ff7021a6807de84b5750e4fa78635051b6` -[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-server-linux-arm64.tar.gz) | `171f15aa8b7c365f4fee70ce025c882a921d0075bd726a99b5534cadd09273ef` -[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-server-linux-ppc64le.tar.gz) | `abc2003d58bd1aca517415c582ed1e8bb1ed596bf04197f4fc7c0c51865a9f86` -[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-server-linux-s390x.tar.gz) | `e2ce834abb4d45d91fd7a8d774e47f0f8092eb4edcf556605c2ef6e2b190b8b1` +[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-server-linux-amd64.tar.gz) | `a8e3d457e5bcc1c09eeb66111e8dd049d6ba048c3c0fa90a61814291afdcde93f1c6dbb07beef090d1d8a9958402ff843e9af23ae9f069c17c0a7c6ce4034686` +[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-server-linux-arm.tar.gz) | `4e17494767000256775e4dd33c0a9b2d152bd4b5fba9f343b6dfeb5746ff34e400a8e0aaf2153476453225ef57e4bb1ae3635416ab18f9e4dabf4e5cc82f8aaa` +[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-server-linux-arm64.tar.gz) | `0ddd0cf0ff56cebfa89efb1972cc2bc6916e824c2af56cfd330ac5638c8918eaf3c60d05714b220dbf4f896160eded123beeba42f5be55fe434a43d04508d86a` +[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-server-linux-ppc64le.tar.gz) | `b93828560224e812ed21b57fea5458fa8560745cfec96fc1677b258393c00e208ad9b99467b575e74e01699ffd75f03f5793675032e7306cba7208c1afb53c8d` +[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-server-linux-s390x.tar.gz) | `154d565329d5ba52cdb7c3d43d8854b7a9b8e34803c4df6b3e6ae74c1a6e255c78e6559b7546b9158df0e3f7931bbdaf43407d95cd875c79f5cce960bb9882dd` ### Node Binaries -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-node-linux-amd64.tar.gz) | `6016c3a1e14c42dcc88caed6497de1b2c56a02bb52d836b19e2ff52098302dda` -[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-node-linux-arm.tar.gz) | `e712e38c8037159ea074ad93c2f2905cf279f3f119e5fdbf9b97391037a8813f` -[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-node-linux-arm64.tar.gz) | `7f4095f12d8ad9438919fa447360113799f88bb9435369b9307a41dd9c7692a6` -[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-node-linux-ppc64le.tar.gz) | `4aeb5dbb0c68e54570542eb5a1d7506d73c81b57eba3c2080ee73bb53dbc3be0` -[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-node-linux-s390x.tar.gz) | `a160599598167208286db6dc73b415952836218d967fa964fc432b213f1b9908` -[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.12.0-rc.2/kubernetes-node-windows-amd64.tar.gz) | `174bedf62b7959d4cb1b1595666f607cd6377c7a2e2208fef5bd554603db5db3` +[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-node-linux-amd64.tar.gz) | `9d18ba5f0c3b09edcf29397a496a1e908f4906087be3792989285630d7bcbaf6cd3bdd7b07dace439823885acc808637190f5eaa240b7b4580acf277b67bb553` +[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-node-linux-arm.tar.gz) | `959b04ff7b8690413e01bffeabaab2119794dedf06b7aae1743e49988f797cb7e6ff12e1a91af2d4c5f664414f3aa4bd9020521c6a21c1196c194d12a6f7fe08` +[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-node-linux-arm64.tar.gz) | `b5c18e8c9e28cf276067c871446720d86b6f162e22c3a5e9343cdbc6857baa6961d09a6908b6acd1bbd132c2e2e526377676babf77b8d3bfb36f8711827c105a` +[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-node-linux-ppc64le.tar.gz) | `63e3504d3b115fdf3396968afafd1107b98e5a1a15b7c042a87f5a9cffbdc274f7b06b07ce90eb51876cfffd57cf7f20180bad7e9f9762af577e51f4f13d2f7a` +[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-node-linux-s390x.tar.gz) | `21c5c2721febf7fddeada9569f3ecbd059267e5d2cc325d98fb74faf1ae9e9e15899750225a1fc7c25feef96e7705b1456cb489f4882b9eb10e78bd0f590d019` +[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.13.0/kubernetes-node-windows-amd64.tar.gz) | `3e73d3ecff14b4c85a71bb6cf91b1ab7d9c3075c64bd5ce6863562ab17bf808b0cbc33ddd25346d25040649c1ad89745796afd218190886b54f1d8acc17896e4` -## Changelog since v1.12.0-rc.1 +# Kubernetes 1.13 Release Notes + +## Security Content + +- CVE-2018-1002105, a critical security issue in the Kubernetes API Server, is resolved in v1.13.0 (and in [v1.10.11](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.10.md/#v11011), [v1.11.5](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md/#v1115), and [v1.12.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md/#v1123)). We recommend all clusters running previous versions update to one of these releases immediately. See issue [#71411](https://github.com/kubernetes/kubernetes/issues/71411) for details. + +## Urgent Upgrade Notes + +### (No, really, you MUST do this before you upgrade) + +Before upgrading to Kubernetes 1.13, you must keep the following in mind: + +- kube-apiserver + - The deprecated `etcd2` storage backend has been removed. Before upgrading a kube-apiserver using `--storage-backend=etcd2`, etcd v2 data must be migrated to the v3 storage backend, and kube-apiserver invocations changed to use `--storage-backend=etcd3`. Please consult the installation procedure used to set up etcd for specific migration instructions. Backups prior to upgrade are always a good practice, but since the etcd2 to etcd3 migration is not reversible, an etcd backup prior to migration is essential. + - The deprecated `--etcd-quorum-read` flag has been removed. Quorum reads are now always enabled when fetching data from etcd. Remove the `--etcd-quorum-read` flag from kube-apiserver invocations before upgrading. +- kube-controller-manager + - The deprecated `--insecure-experimental-approve-all-kubelet-csrs-for-group` flag has been removed. +- kubelet + - The deprecated `--google-json-key` flag has been removed. Remove the `--google-json-key` flag from kubelet invocations before upgrading. ([#69354](https://github.com/kubernetes/kubernetes/pull/69354), [@yujuhong](https://github.com/yujuhong)) + - DaemonSet pods now make use of scheduling features that require kubelets to be at 1.11 or above. Ensure all kubelets in the cluster are at 1.11 or above before upgrading kube-controller-manager to 1.13. + - The schema for the alpha `CSINodeInfo` CRD has been split into `spec` and `status` fields, and new fields `status.available` and `status.volumePluginMechanism` added. Clusters using the previous alpha schema must delete and recreate the CRD using the new schema. ([#70515](https://github.com/kubernetes/kubernetes/pull/70515), [@davidz627](https://github.com/davidz627)) +- kube-scheduler dropped support for configuration files with apiVersion `componentconfig/v1alpha1`. Ensure kube-scheduler is configured using command-line flags or a configuration file with apiVersion `kubescheduler.config.k8s.io/v1alpha1` before upgrading to 1.13. +- kubectl + - The deprecated command `run-container` has been removed. Invocations should use `kubectl run` instead ([#70728](https://github.com/kubernetes/kubernetes/pull/70728), [@Pingan2017](https://github.com/Pingan2017)) +- client-go releases will no longer have bootstrap (k8s.io/client-go/tools/bootstrap) related code. Any reference to it will break. Please redirect all references to k8s.io/bootstrap instead. ([#67356](https://github.com/kubernetes/kubernetes/pull/67356), [@yliaog](https://github.com/yliaog)) +- Kubernetes cannot distinguish between GCE Zonal PDs and Regional PDs with the same name. To workaround this issue, precreate PDs with unique names. PDs that are dynamically provisioned do not encounter this issue. ([#70716](https://github.com/kubernetes/kubernetes/pull/70716), [@msau42](https://github.com/msau42)) + +## Known Issues + +- If kubelet plugin registration for a driver fails, kubelet will not retry. The driver must delete and recreate the driver registration socket in order to force kubelet to attempt registration again. Restarting only the driver container may not be sufficient to trigger recreation of the socket, instead a pod restart may be required. ([#71487](https://github.com/kubernetes/kubernetes/issues/71487)) +- In some cases, a Flex volume resize may leave a PVC with erroneous Resizing condition even after volume has been successfully expanded. Users may choose to delete the condition, but it is not required. ([#71470](https://github.com/kubernetes/kubernetes/issues/71470)) +- The CSI driver-registrar external sidecar container v1.0.0-rc2 is known to take up to 1 minute to start in some cases. We expect this issue to be resolved in a future release of the sidecar container. For verification, please see the release notes of future releases of the external sidecar container. ([#76](https://github.com/kubernetes-csi/driver-registrar/issues/76)) +- When using IPV6-only, be sure to use `proxy-mode=iptables` as `proxy-mode=ipvs` is known to not work. ([#68437](https://github.com/kubernetes/kubernetes/issues/68437)) + +## Deprecations + +- kube-apiserver + - The `--service-account-api-audiences` flag is deprecated in favor of `--api-audiences`. The old flag is accepted with a warning but will be removed in a future release. ([#70105](https://github.com/kubernetes/kubernetes/pull/70105), [@mikedanese](https://github.com/mikedanese)) + - The `--experimental-encryption-provider-config` flag is deprecated in favor of `--encryption-provider-config`. The old flag is accepted with a warning but will be removed in 1.14. ([#71206](https://github.com/kubernetes/kubernetes/pull/71206), [@stlaz](https://github.com/stlaz)) + - As part of graduating the etcd encryption feature to beta, the configuration file referenced by `--encryption-provider-config` now uses `kind: EncryptionConfiguration` and `apiVersion: apiserver.config.k8s.io/v1`. Support for `kind: EncryptionConfig` and `apiVersion: v1` is deprecated and will be removed in a future release. ([#67383](https://github.com/kubernetes/kubernetes/pull/67383), [@stlaz](https://github.com/stlaz)) + - The `--deserialization-cache-size` flag is deprecated, and will be removed in a future release. The flag is inactive since the etcd2 storage backend was removed. ([#69842](https://github.com/kubernetes/kubernetes/pull/69842), [@liggitt](https://github.com/liggitt)) + - The `Node` authorization mode no longer allows kubelets to delete their Node API objects (prior to 1.11, in rare circumstances related to cloudprovider node ID changes, kubelets would attempt to delete/recreate their Node object at startup) ([#71021](https://github.com/kubernetes/kubernetes/pull/71021), [@liggitt](https://github.com/liggitt)) + - The built-in `system:csi-external-provisioner` and `system:csi-external-attacher` cluster roles are deprecated and will not be auto-created in a future release. CSI deployments should provide their own RBAC role definitions with required permissions. ([#69868](https://github.com/kubernetes/kubernetes/pull/69868), [@pohly]( https://github.com/pohly)) + - The built-in `system:aws-cloud-provider` cluster role is deprecated and will not be auto-created in a future release. Deployments using the AWS cloud provider should grant required permissions to the `aws-cloud-provider` service account in the `kube-system` namespace as part of deployment. ([#66635](https://github.com/kubernetes/kubernetes/pull/66635), [@wgliang](https://github.com/wgliang)) +- kubelet + - Use of the beta plugin registration directory `{kubelet_root_dir}/plugins/` for registration of external drivers via the kubelet plugin registration protocol is deprecated in favor of `{kubelet_root_dir}/plugins_registry/`. Support for the old directory is planned to be removed in v1.15. Device plugin and CSI storage drivers should switch to the new directory prior to v1.15. Only CSI storage drivers that support 0.x versions of the CSI API are allowed in the old directory. ([#70494](https://github.com/kubernetes/kubernetes/pull/70494) by [@RenaudWasTaken](https://github.com/RenaudWasTaken) and [#71314](https://github.com/kubernetes/kubernetes/pull/71314) by [@saad-ali](https://github.com/saad-ali)) + - With the release of the CSI 1.0 API, support for CSI drivers using 0.3 and older releases of the CSI API is deprecated, and is planned to be removed in Kubernetes v1.15. CSI drivers should be updated to support the CSI 1.0 API, and deployed in the new kubelet plugin registration directory (`{kubelet_root_dir}/plugins_registry/`) once all nodes in the cluster are at 1.13 or higher ([#71020](https://github.com/kubernetes/kubernetes/pull/71020) and [#71314](https://github.com/kubernetes/kubernetes/pull/71314), both by [@saad-ali](https://github.com/saad-ali)) + - Use of the `--node-labels` flag to set labels under the `kubernetes.io/` and `k8s.io/` prefix will be subject to restriction by the `NodeRestriction` admission plugin in future releases. [See admission plugin documentation](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#noderestriction) for allowed labels. ([#68267](https://github.com/kubernetes/kubernetes/pull/68267), [@liggitt](https://github.com/liggitt)) +- kube-scheduler + - The alpha critical pod annotation (`scheduler.alpha.kubernetes.io/critical-pod`) is deprecated. Pod priority should be used instead to mark pods as critical. ([#70298](https://github.com/kubernetes/kubernetes/pull/70298), [@bsalamat](https://github.com/bsalamat)) +- The following features are now GA, and the associated feature gates are deprecated and will be removed in a future release: + - CSIPersistentVolume + - GCERegionalPersistentDisk + - KubeletPluginsWatcher + - VolumeScheduling +- kubeadm + - The DynamicKubeletConfig feature gate is deprecated. The functionality is still accessible by using the kubeadm alpha kubelet enable-dynamic command. + - The command `kubeadm config print-defaults` is deprecated in favor of `kubeadm config print init-defaults` and `kubeadm config print join-defaults` ([#69617](https://github.com/kubernetes/kubernetes/pull/69617), [@rosti](https://github.com/rosti)) + - support for the `v1alpha3` configuration file format is deprecated and will be removed in 1.14. Use `kubeadm config migrate` to migrate `v1alpha3` configuration files to `v1beta1`, which provides improvements in image repository management, addons configuration, and other areas. The documentation for `v1beta1` can be found here: https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1 +- The `node.status.volumes.attached.devicePath` field is deprecated for CSI volumes and will not be set in future releases ([#71095](https://github.com/kubernetes/kubernetes/pull/71095), [@msau42](https://github.com/msau42)) +- kubectl + - The `kubectl convert` command is deprecated and will be removed in a future release ([#70820](https://github.com/kubernetes/kubernetes/pull/70820), [@seans3](https://github.com/seans3)) +- Support for passing unknown provider names to the E2E test binaries is deprecated and will be removed in a future release. Use `--provider=skeleton` (no ssh access) or `--provider=local` (local cluster with ssh) instead. ([#70141](https://github.com/kubernetes/kubernetes/pull/70141), [@pohly](https://github.com/pohly)) + +## Major Themes + +### SIG API Machinery + +For the 1.13 release, SIG API Machinery is happy to announce that the [dry-run functionality](https://kubernetes.io//docs/reference/using-api/api-concepts/#dry-run) is now beta. + +### SIG Auth + +With this release we've made several important enhancements to core SIG Auth areas. In the authorization category, we've further reduced Kubelet privileges by [restricting node self-updates of labels to a whitelisted selection and by disallowing kubelets from deleting their Node API object](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#noderestriction). In authentication, we added alpha-level support for automounting improved service account tokens through projected volumes. We also enabled [audience validation in TokenReview](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#tokenreview-v1-authentication-k8s-io) for the new tokens for improved scoping. Under audit logging, the new alpha-level "dynamic audit configuration" adds support for [dynamically registering webhooks to receive a stream of audit events](https://kubernetes.io/docs/tasks/debug-application-cluster/audit/#dynamic-backend). Finally, we've enhanced secrets protection by graduating [etcd encryption](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/) out of experimental. + +### SIG AWS + +In v1.13 we worked on tighter integrations of Kubernetes API objects with AWS services. These include three out-of-tree alpha feature releases: + +1) Alpha for AWS ALB (Application Load Balancer) integration to Kubernetes Ingress resources. +2) Alpha for CSI specification 0.3 integration to AWS EBS (Elastic Block Store) +3) Alpha for the cloudprovider-aws cloud controller manager binary. Additionally we added [aws-k8s-tester](https://github.com/kubernetes/test-infra/issues/9814), deployer interface for kubetest, to the test-infra repository. This plugin allowed us to integrate Prow to the 3 subprojects defined above in order to provide CI signal for all 3 features. The CI signal is visible [here](https://testgrid.k8s.io/) under SIG-AWS. + +For detailed release notes on the three alpha features from SIG AWS, please refer to the following Changelogs: + +- [aws-alb-ingress-controller v1.0.0](https://github.com/kubernetes-sigs/aws-alb-ingress-controller/releases/tag/v1.0.0) +- [aws-ebs-csi-driver v0.1](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/CHANGELOG-0.1.md) +- [cloudprovider-aws external v0.1.0] (https://github.com/kubernetes/cloud-provider-aws/blob/master/changelogs/CHANGELOG-0.1.md) + +### SIG Azure + +For 1.13 SIG Azure was focused on adding additional Azure Disk support for Ultra SSD, Standard SSD, and Premium Azure Files. Azure Availability Zones and cross resource group nodes were also moved from Alpha to Beta in 1.13. + +### SIG Big Data + +During the 1.13 release cycle, SIG Big Data has been focused on community engagements relating to 3rd-party project integrations with Kubernetes. There have been no impacts on the 1.13 release. + +### SIG CLI + +Over the course of 1.13 release SIG CLI mostly focused on stabilizing the items we’ve been working on over the past releases such as server-side printing and its support in kubectl, as well as finishing [kubectl diff which is based on server-side dry-run feature](https://kubernetes.io/docs/concepts/overview/object-management-kubectl/#how-to-create-objects). We’ve continued separating kubectl code to prepare for extraction out of main repository. Finally, thanks to the awesome support and feedback from community we’ve managed to promote the new [plugin mechanism to Beta](https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/). + +### SIG Cloud Provider + +For v1.13, SIG Cloud Provider has been focused on stabilizing the common APIs and interfaces consumed by cloud providers today. This involved auditing the cloud provider APIs for anything that should be deprecated as well as adding changes where necessary. In addition, SIG Cloud Provider has begun exploratory work around having a “cloud provider” e2e test suite which can be used to test common cloud provider functionalities with resources such as nodes and load balancers. + +We are also continuing our long running effort to extract all the existing cloud providers that live in k8s.io/kubernetes into their own respective repos. Along with this migration, we are slowly transitioning users to use the cloud-controller-manager for any cloud provider features instead of the kube-controller-manager. + +### SIG Cluster Lifecycle + +For 1.13 SIG Cluster Lifecycle is pleased to announce the long awaited promotion of kubeadm to stable GA, and the promotion of kubeadm’s configuration API to `v1beta1`. +In this release the SIG again focused on further improving the user experience on cluster creation and also fixing a number of bugs and other assorted improvements. + +Some notable changes in kubeadm since Kubernetes 1.12: + +- kubeadm’s configuration API is now `v1beta1`. The new configuration format provides improvements in - image repository management, addons configuration, and other areas. We encourage `v1alpha3` users to migrate to this configuration API using `kubeadm config migrate`, as `v1alpha3` will be removed in 1.14. The documentation for `v1beta1` can be found here: https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1 +- kubeadm has graduated `kubeadm alpha phase` commands to `kubeadm init phase`. This means that the phases of creating a control-plane node are now tightly integrated as part of the `init` command. Alpha features, not yet ready for GA are still kept under `kubeadm alpha` and we appreciate feedback on them. +- `kubeadm init` and `kubeadm init phase` now have a `--image-repository` flag, improving support for environments with limited access to official kubernetes repository. +- The DynamicKubeletConfig and SelfHosting functionality was moved outside of `kubeadm init` and feature gates and is now exposed under `kubeadm alpha`. +- Kubeadm init phase certs now support the `--csr-only` option, simplifying custom CA creation. +- `kubeadm join --experimental-control-plane` now automatically adds a new etcd member for `local etcd` mode, further simplifying required tasks for HA clusters setup. +- Improvements were made to `kubeadm reset` related to cleaning etcd and notifying the user about the state of iptables. +- kubeadm commands now print warnings if input YAML documents contain unknown or duplicate fields. +- kubeadm now properly recognizes Docker 18.09.0 and newer, but still treats 18.06 as the default supported version. +- kubeadm now automatically sets the `--pod-infra-container-image` flag when starting the kubelet. + +### SIG IBM Cloud + +The IBM Cloud SIG was focused on defining its charter and working towards moving its cloud provider code to an external repository with a goal to have this work done by the end of Kubernetes 1.14 release cycle. In the SIG meetings, we also made sure to share updates on the latest Kubernetes developments in the IBM Cloud like the availability of Kubernetes v1.12.2 in the IBM Cloud Kubernetes Service (IKS). The SIG updates were provided in the Kubernetes community weekly call and at the KubeCon China 2018. + +### SIG Multicluster + +Moving Federation v2 from Alpha towards Beta has been the focus of our effort over the past quarter. To this end we engaged with end users, and successfully enlisted additional contributors from companies including IBM, Amadeus, Cisco and others. Federation v2 provides a suite of decoupled API’s and re-usable components for building multi-cluster control planes. We plan to start releasing Beta components in late 2018. In addition, more minor updates were made to our cluster-registry and multi-cluster ingress sub-projects. + +### SIG Network + +For 1.13, the areas of focus were in IPv6, DNS improvements and some smaller items: +CoreDNS is now the default cluster DNS passing all of the scale/resource usage tests +Node-local DNS cache feature is available in Alpha. This feature deploys a lightweight DNS caching Daemonset that avoids the conntrack and converts queries from UDP to more reliable TCP. +PodReady++ feature now has `kubectl` CLI support. + +Progress was made towards finalizing the IPv6 dual stack support KEP and support for topological routing of services. + +### SIG Node + +SIG Node focused on stability and performance improvements in the 1.13 release. A new alpha feature is introduced to improve the mechanism that nodes heartbeat back to the control plane. The `NodeLease` feature results in the node using a `Lease` resource in the `kube-node-lease` namespace that is renewed periodically. The `NodeStatus` that was used previously to heartbeat back to the control plane is only updated when it changes. This reduces load on the control plane for large clusters. The Kubelet plugin registration mechanism, which enables automatic discovery of external plugins (including CSI and device plugins) has been promoted to stable in this release (introduced as alpha in 1.11 and promoted to beta in 1.12). + +### SIG Openstack + +The major theme for the SIG OpenStack release is the work-in-progress for removing the in-tree provider. This work, being done in conjunction with SIG Cloud Provider, is focusing on moving internal APIs that the OpenStack (and other providers) depends upon to staging to guarantee API stability. This work also included abstracting the in-tree Cinder API and refactoring code to the external Cinder provider to remove additional Cinder volume provider code. + +Additional work was also done to implement an OpenStack driver for the Cluster API effort lead by SIG Cluster Lifecycle. For the external Cloud-Provider-OpenStack code, the SIG largely focused on bug fixes and updates to match K8s 1.13 development. + +### SIG Scalability + +SIG Scalability has mostly focused on stability and deflaking our tests, investing into framework for writing scalability tests (ClusterLoader v2) with a goal to migrate all tests to it by the end of 2018 and on the work towards extending definition of Kubernetes scalability by providing more/better user-friendly SLIs/SLOs. + +### SIG Scheduling + +SIG Scheduling has mostly focused on stability in 1.13 and has postponed some of the major features to the next versions. There are still two notable changes: 1. TaintBasedEviction is moved to Beta and will be enabled by default. With this feature enabled, condition taints are automatically added to the nodes and pods can add tolerations for them if needed. 2. Pod critical annotation is deprecated. Pods should use pod priority instead of the annotation. + +It is worth noting again that kube-scheduler will use apiVersion `kubescheduler.config.k8s.io/v1alpha1` instead of `componentconfig/v1alpha1` in its configuration files in 1.13. + +### SIG Service Catalog + +The Service Plan Defaults feature is still under active development. +We continue to improve the UX for the svcat CLI, specifically filling in gaps for the new Namespaced Service Broker feature. + +### SIG Storage + +Over the last year, SIG Storage has been focused on adding support for the Container Storage Interface (CSI) to Kubernetes. The specification recently moved to 1.0, and on the heels of this achievement, Kubernetes v1.13 moves CSI support for PersistentVolumes to GA. + +With CSI the Kubernetes volume layer becomes truly extensible, allowing third party storage developers to write drivers making their storage systems available in Kubernetes without having to touch the core code. + +CSI was first introduction as alpha in Kubernetes v1.9 and moved to beta in Kubernetes v1.10. + +You can find a list of sample and production drivers in the [CSI Documentation](https://kubernetes.io/docs/concepts/storage/volumes/#csi). + +SIG Storage also moves support for Block Volumes to beta (introduced as alpha in v1.9) and support for Topology Aware Volume Scheduling to stable (introduced as alpha in v1.9 and promoted to beta in 1.10). + +### SIG UI + +The migration to the newest version of Angular is still under active development as it is most important thing on the roadmap at the moment. We are getting closer to to the new release. We continue fixing bugs and adding other improvements. + +### SIG VMWare + +Major focus for SIG VMware for this release is the work on moving internal APIs that the vSphere provider depends upon to staging to guarantee API stability. This work is being done in conjunction with SIG Cloud Provider and includes the creation of a brand new vsphere-csi plugin to replace the current volume functionalities in-tree. + +Additional work was also done to implement a vSphere provider for the Cluster API effort lead by SIG Cluster Lifecycle. For the out-of-tree vSphere cloud provider, the SIG largely focused on bug fixes and updates to match K8s 1.13 development. + +### SIG Windows + +SIG Windows focused on improving reliability for Windows and Kubernetes support + +## New Features + +- kubelet: When node lease feature is enabled, kubelet reports node status to api server only if there is some change or it didn't report over last report interval. ([#69753](https://github.com/kubernetes/kubernetes/pull/69753), [@wangzhen127](https://github.com/wangzhen127)) +- vSphereVolume implements Raw Block Volume Support ([#68761](https://github.com/kubernetes/kubernetes/pull/68761), [@fanzhangio](https://github.com/fanzhangio)) +- CRD supports multi-version Schema, Subresources and AdditionalPrintColumns (NOTE that CRDs created prior to 1.13 populated the top-level additionalPrinterColumns field by default. To apply an updated that changes to per-version additionalPrinterColumns, the top-level additionalPrinterColumns field must be explicitly set to null). ([#70211](https://github.com/kubernetes/kubernetes/pull/70211), [@roycaihw](https://github.com/roycaihw)) +- New addon in addon manager that automatically installs CSI CRDs if CSIDriverRegistry or CSINodeInfo feature gates are true. ([#70193](https://github.com/kubernetes/kubernetes/pull/70193), [@saad-ali](https://github.com/saad-ali)) +- Delegated authorization can now allow unrestricted access for `system:masters` like the main kube-apiserver ([#70671](https://github.com/kubernetes/kubernetes/pull/70671), [@deads2k](https://github.com/deads2k)) +- Added dns capabilities for Windows CNI plugins: ([#67435](https://github.com/kubernetes/kubernetes/pull/67435), [@feiskyer](https://github.com/feiskyer)) +- kube-apiserver: `--audit-webhook-version` and `--audit-log-version` now default to `audit.k8s.io/v1` if unspecified ([#70476](https://github.com/kubernetes/kubernetes/pull/70476), [@charrywanganthony](https://github.com/charrywanganthony)) +- kubeadm: timeoutForControlPlane is introduced as part of the API Server config, that controls the timeout for the wait for control plane to be up. Default value is 4 minutes. ([#70480](https://github.com/kubernetes/kubernetes/pull/70480), [@rosti](https://github.com/rosti)) +- `--api-audiences` now defaults to the `--service-account-issuer` if the issuer is provided but the API audience is not. ([#70308](https://github.com/kubernetes/kubernetes/pull/70308), [@mikedanese](https://github.com/mikedanese)) +- Added support for projected volume in describe function ([#70158](https://github.com/kubernetes/kubernetes/pull/70158), [@WanLinghao](https://github.com/WanLinghao)) +- kubeadm now automatically creates a new stacked etcd member when joining a new control plane node (does not applies to external etcd) ([#69486](https://github.com/kubernetes/kubernetes/pull/69486), [@fabriziopandini](https://github.com/fabriziopandini)) +- Display the usage of ephemeral-storage when using `kubectl describe node` ([#70268](https://github.com/kubernetes/kubernetes/pull/70268), [@Pingan2017](https://github.com/Pingan2017)) +- Added functionality to enable br_netfilter and ip_forward for debian packages to improve kubeadm support for CRI runtime besides Docker. ([#70152](https://github.com/kubernetes/kubernetes/pull/70152), [@ashwanikhemani](https://github.com/ashwanikhemani)) +- Added regions ap-northeast-3 and eu-west-3 to the list of well known AWS regions. ([#70252](https://github.com/kubernetes/kubernetes/pull/70252), [@nckturner](https://github.com/nckturner)) +- kubeadm: Implemented preflight check to ensure that number of CPUs ([#70048](https://github.com/kubernetes/kubernetes/pull/70048), [@bart0sh](https://github.com/bart0sh)) +- CoreDNS is now the default DNS server in kube-up deployments. ([#69883](https://github.com/kubernetes/kubernetes/pull/69883), [@chrisohaver](https://github.com/chrisohaver)) +- Opt out of chowning and chmoding from kubectl cp. ([#69573](https://github.com/kubernetes/kubernetes/pull/69573), [@bjhaid](https://github.com/bjhaid)) +- Failed to provision volume with StorageClass "azurefile-premium": failed to create share andy-mg1121-dynamic-pvc-1a7b2813-d1b7-11e8-9e96-000d3a03e16b in account f7228f99bcde411e8ba4900: failed to create file share, err: storage: service returned error: StatusCode=400, ErrorCode=InvalidHeaderValue, ErrorMessage=The value for one of the HTTP headers is not in the correct format. ([#69718](https://github.com/kubernetes/kubernetes/pull/69718), [@andyzhangx](https://github.com/andyzhangx)) +- `TaintBasedEvictions` feature is promoted to beta. ([#69824](https://github.com/kubernetes/kubernetes/pull/69824), [@Huang-Wei](https://github.com/Huang-Wei)) +- Fixed https://github.com/kubernetes/client-go/issues/478 by adding support for JSON Patch in client-go/dynamic/fake ([#69330](https://github.com/kubernetes/kubernetes/pull/69330), [@vaikas-google](https://github.com/vaikas-google)) +- Dry-run is promoted to Beta and will be enabled by default. ([#69644](https://github.com/kubernetes/kubernetes/pull/69644), [@apelisse](https://github.com/apelisse)) +- `kubectl get priorityclass` now prints value column by default. ([#69431](https://github.com/kubernetes/kubernetes/pull/69431), [@Huang-Wei](https://github.com/Huang-Wei)) +- Added a new container based image for running e2e tests ([#69368](https://github.com/kubernetes/kubernetes/pull/69368), [@dims](https://github.com/dims)) +- The `LC_ALL` and `LC_MESSAGES` env vars can now be used to set desired locale for `kubectl` while keeping `LANG` unchanged. ([#69500](https://github.com/kubernetes/kubernetes/pull/69500), [@m1kola](https://github.com/m1kola)) +- NodeLifecycleController: Now node lease renewal is treated as the heartbeat signal from the node, in addition to NodeStatus Update. ([#69241](https://github.com/kubernetes/kubernetes/pull/69241), [@wangzhen127](https://github.com/wangzhen127)) +- Added dynamic shared informers to write generic, non-generated controllers ([#69308](https://github.com/kubernetes/kubernetes/pull/69308), [@p0lyn0mial](https://github.com/p0lyn0mial)) +- Upgraded to etcd 3.3 client ([#69322](https://github.com/kubernetes/kubernetes/pull/69322), [@jpbetz](https://github.com/jpbetz)) +- It is now possible to use named ports in the `kubectl port-forward` command ([#69477](https://github.com/kubernetes/kubernetes/pull/69477), [@m1kola](https://github.com/m1kola)) +- `kubectl wait` now supports condition value checks other than true using `--for condition=available=false` ([#69295](https://github.com/kubernetes/kubernetes/pull/69295), [@deads2k](https://github.com/deads2k)) +- Updated defaultbackend image to 1.5. Users should concentrate on updating scripts to the new version. ([#69120](https://github.com/kubernetes/kubernetes/pull/69120), [@aledbf](https://github.com/aledbf)) +- Bumped Dashboard version to v1.10.0 ([#68450](https://github.com/kubernetes/kubernetes/pull/68450), [@jeefy](https://github.com/jeefy)) +- Added env variables to control CPU requests of kube-controller-manager and kube-scheduler. ([#68823](https://github.com/kubernetes/kubernetes/pull/68823), [@loburm](https://github.com/loburm)) +- PodSecurityPolicy objects now support a `MayRunAs` rule for `fsGroup` and `supplementalGroups` options. This allows specifying ranges of allowed GIDs for pods/containers without forcing a default GID the way `MustRunAs` does. This means that a container to which such a policy applies to won't use any fsGroup/supplementalGroup GID if not explicitly specified, yet a specified GID must still fall in the GID range according to the policy. ([#65135](https://github.com/kubernetes/kubernetes/pull/65135), [@stlaz](https://github.com/stlaz)) +- Upgrade Stackdriver Logging Agent addon image to 0.6-1.6.0-1 to use Fluentd v1.2. This provides nanoseconds timestamp granularity for logs. ([#70954](https://github.com/kubernetes/kubernetes/pull/70954), [@qingling128](https://github.com/qingling128)) +- When the BoundServiceAccountTokenVolumes Alpha feature is enabled, ServiceAccount volumes now use a projected volume source and their names have the prefix "kube-api-access". ([#69848](https://github.com/kubernetes/kubernetes/pull/69848), [@mikedanese](https://github.com/mikedanese)) +- Raw block volume support is promoted to beta, and enabled by default. This is accessible via the `volumeDevices` container field in pod specs, and the `volumeMode` field in persistent volume and persistent volume claims definitions. ([#71167](https://github.com/kubernetes/kubernetes/pull/71167), [@msau42](https://github.com/msau42)) +- TokenReview now supports audience validation of tokens with audiences other than the kube-apiserver. ([#62692](https://github.com/kubernetes/kubernetes/pull/62692), [@mikedanese](https://github.com/mikedanese)) +- StatefulSet is supported in `kubectl autoscale` command ([#71103](https://github.com/kubernetes/kubernetes/pull/71103), [@Pingan2017](https://github.com/Pingan2017)) +- Kubernetes v1.13 moves support for Container Storage Interface to GA. As part of this move Kubernetes now supports CSI v1.0.0 and deprecates support for CSI 0.3 and older releases. Older CSI drivers must be updated to CSI 1.0 and moved to the new kubelet plugin registration directory in order to work with Kubernetes 1.15+. ([#71020](https://github.com/kubernetes/kubernetes/pull/71020), [@saad-ali](https://github.com/saad-ali)) +- Added option to create CSRs instead of certificates for kubeadm init phase certs and kubeadm alpha certs renew ([#70809](https://github.com/kubernetes/kubernetes/pull/70809), [@liztio](https://github.com/liztio)) +- Added a kubelet socket which serves an grpc service containing the devices used by containers on the node. ([#70508](https://github.com/kubernetes/kubernetes/pull/70508), [@dashpole](https://github.com/dashpole)) +- Added DynamicAuditing feature which allows for the configuration of audit webhooks through the use of an AuditSink API object. ([#67257](https://github.com/kubernetes/kubernetes/pull/67257), [@pbarker](https://github.com/pbarker)) +- The kube-apiserver's healthz now takes in an optional query parameter which allows you to disable health checks from causing healthz failures. ([#70676](https://github.com/kubernetes/kubernetes/pull/70676), [@logicalhan](https://github.com/logicalhan)) +- Introduced support for running a nodelocal dns cache. It is disabled by default, can be enabled by setting KUBE_ENABLE_NODELOCAL_DNS=true ([#70555](https://github.com/kubernetes/kubernetes/pull/70555), [@prameshj](https://github.com/prameshj)) +- Added readiness gates in extended output for pods ([#70775](https://github.com/kubernetes/kubernetes/pull/70775), [@freehan](https://github.com/freehan)) +- Added `Ready` column and improve human-readable output of Deployments and StatefulSets ([#70466](https://github.com/kubernetes/kubernetes/pull/70466), [@Pingan2017](https://github.com/Pingan2017)) +- Added `kubelet_container_log_size_bytes` metric representing the log file size of a container. ([#70749](https://github.com/kubernetes/kubernetes/pull/70749), [@brancz](https://github.com/brancz)) +- NodeLifecycleController: When node lease feature is enabled, node lease will be deleted when the corresponding node is deleted. ([#70034](https://github.com/kubernetes/kubernetes/pull/70034), [@wangzhen127](https://github.com/wangzhen127)) +- GCERegionalPersistentDisk feature is GA now! ([#70716](https://github.com/kubernetes/kubernetes/pull/70716), [@jingxu97](https://github.com/jingxu97)) +- Added secure port 10259 to the kube-scheduler (enabled by default) and deprecate old insecure port 10251. Without further flags self-signed certs are created on startup in memory. ([#69663](https://github.com/kubernetes/kubernetes/pull/69663), [@sttts](https://github.com/sttts)) + +## Release Notes From SIGs + +### SIG API Machinery + +- The OwnerReferencesPermissionEnforcement admission plugin now checks authorization for the correct scope (namespaced or cluster-scoped) of the owner resource type. Previously, it always checked permissions at the same scope as the child resource. ([#70389](https://github.com/kubernetes/kubernetes/pull/70389), [@caesarxuchao](https://github.com/caesarxuchao)) +- OpenAPI spec now correctly marks delete request's body parameter as optional ([#70032](https://github.com/kubernetes/kubernetes/pull/70032), [@iamneha](https://github.com/iamneha)) +- The rules for incrementing `metadata.generation` of custom resources changed: ([#69059](https://github.com/kubernetes/kubernetes/pull/69059), [@caesarxuchao](https://github.com/caesarxuchao)) + - If the custom resource participates the spec/status convention, the metadata.generation of the CR increments when there is any change, except for the changes to the metadata or the changes to the status. + - If the custom resource does not participate the spec/status convention, the metadata.generation of the CR increments when there is any change to the CR, except for changes to the metadata. + - A custom resource is considered to participate the spec/status convention if and only if the "CustomResourceSubresources" feature gate is turned on and the CRD has `.spec.subresources.status={}`. +- Fixed patch/update operations on multi-version custom resources ([#70087](https://github.com/kubernetes/kubernetes/pull/70087), [@liggitt](https://github.com/liggitt)) +- Reduced memory utilization of admission webhook metrics by removing resource related labels. ([#69895](https://github.com/kubernetes/kubernetes/pull/69895), [@jpbetz](https://github.com/jpbetz)) +- Kubelet can now parse PEM file containing both TLS certificate and key in arbitrary order. Previously key was always required to be first. ([#69536](https://github.com/kubernetes/kubernetes/pull/69536), [@awly](https://github.com/awly)) +- Code-gen: Removed lowercasing for project imports ([#68484](https://github.com/kubernetes/kubernetes/pull/68484), [@jsturtevant](https://github.com/jsturtevant)) +- Fixed client cert setup in delegating authentication logic ([#69430](https://github.com/kubernetes/kubernetes/pull/69430), [@DirectXMan12](https://github.com/DirectXMan12)) +- OpenAPI spec and API reference now reflect dryRun query parameter for POST/PUT/PATCH operations ([#69359](https://github.com/kubernetes/kubernetes/pull/69359), [@roycaihw](https://github.com/roycaihw)) +- Fixed the sample-apiserver so that its BanFlunder admission plugin can be used. ([#68417](https://github.com/kubernetes/kubernetes/pull/68417), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) +- APIService availability related to networking glitches are corrected faster ([#68678](https://github.com/kubernetes/kubernetes/pull/68678), [@deads2k](https://github.com/deads2k)) +- Fixed an issue with stuck connections handling error responses ([#71412](https://github.com/kubernetes/kubernetes/pull/71412), [@liggitt](https://github.com/liggitt)) +- apiserver: fixed handling and logging of panics in REST handlers ([#71076](https://github.com/kubernetes/kubernetes/pull/71076), [@liggitt](https://github.com/liggitt)) +- kube-controller-manager no longer removes ownerReferences from ResourceQuota objects ([#70035](https://github.com/kubernetes/kubernetes/pull/70035), [@liggitt](https://github.com/liggitt)) +- "unfinished_work_microseconds" is added to the workqueue metrics; it can be used to detect stuck worker threads. (kube-controller-manager runs many workqueues.) ([#70884](https://github.com/kubernetes/kubernetes/pull/70884), [@lavalamp](https://github.com/lavalamp)) +- Timeouts set in ListOptions for clients are also be respected locally ([#70998](https://github.com/kubernetes/kubernetes/pull/70998), [@deads2k](https://github.com/deads2k)) +- Added support for CRD conversion webhook ([#67006](https://github.com/kubernetes/kubernetes/pull/67006), [@mbohlool](https://github.com/mbohlool)) +- client-go: fixed sending oversized data frames to spdystreams in remotecommand.NewSPDYExecutor ([#70999](https://github.com/kubernetes/kubernetes/pull/70999), [@liggitt](https://github.com/liggitt)) +- Fixed missing flags in `-controller-manager --help`. ([#71298](https://github.com/kubernetes/kubernetes/pull/71298), [@stewart-yu](https://github.com/stewart-yu)) +- Fixed missing flags in `kube-apiserver --help`. ([#70204](https://github.com/kubernetes/kubernetes/pull/70204), [@imjching](https://github.com/imjching)) +- The caBundle and service fields in admission webhook API objects now correctly indicate they are optional ([#70138](https://github.com/kubernetes/kubernetes/pull/70138), [@liggitt](https://github.com/liggitt)) +- Fixed an issue with stuck connections handling error responses ([#71419](https://github.com/kubernetes/kubernetes/pull/71419), [@liggitt](https://github.com/liggitt)) +- kube-controller-manager and cloud-controller-manager now hold generated serving certificates in-memory unless a writeable location is specified with --cert-dir ([#69884](https://github.com/kubernetes/kubernetes/pull/69884), [@liggitt](https://github.com/liggitt)) +- CCM server will not listen insecurely if secure port is specified ([#68982](https://github.com/kubernetes/kubernetes/pull/68982), [@aruneli](https://github.com/aruneli)) +- List operations against the API now return internal server errors instead of partially complete lists when a value cannot be transformed from storage. The updated behavior is consistent with all other operations that require transforming data from storage such as watch and get. ([#69399](https://github.com/kubernetes/kubernetes/pull/69399), [@mikedanese](https://github.com/mikedanese)) + +### SIG Auth + +- API Server can be configured to reject requests that cannot be audit-logged. ([#65763](https://github.com/kubernetes/kubernetes/pull/65763), [@x13n](https://github.com/x13n)) +- Go clients created from a kubeconfig that specifies a TokenFile now periodically reload the token from the specified file. ([#70606](https://github.com/kubernetes/kubernetes/pull/70606), [@mikedanese](https://github.com/mikedanese)) +- When `--rotate-server-certificates` is enabled, kubelet will no longer request a new certificate on startup if the current certificate on disk is satisfactory. ([#69991](https://github.com/kubernetes/kubernetes/pull/69991), [@agunnerson-ibm](https://github.com/agunnerson-ibm)) +- Added dynamic audit configuration api ([#67547](https://github.com/kubernetes/kubernetes/pull/67547), [@pbarker](https://github.com/pbarker)) +- Added ability to control primary GID of containers through Pod Spec and PodSecurityPolicy ([#67802](https://github.com/kubernetes/kubernetes/pull/67802), [@krmayankk](https://github.com/krmayankk)) +- kube-apiserver: the `NodeRestriction` admission plugin now prevents kubelets from modifying `Node` labels prefixed with `node-restriction.kubernetes.io/`. The `node-restriction.kubernetes.io/` label prefix is reserved for cluster administrators to use for labeling `Node` objects to target workloads to nodes in a way that kubelets cannot modify or spoof. ([#68267](https://github.com/kubernetes/kubernetes/pull/68267), [@liggitt](https://github.com/liggitt)) + +### SIG Autoscaling + +- Updated Cluster Autoscaler version to 1.13.0. See the [Release Notes](https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.13.0) for more information. ([#71513](https://github.com/kubernetes/kubernetes/pull/71513), [@losipiuk](https://github.com/losipiuk)) + +### SIG AWS + +- `service.beta.kubernetes.io/aws-load-balancer-internal` now supports true and false values, previously it only supported non-empty strings ([#69436](https://github.com/kubernetes/kubernetes/pull/69436), [@mcrute](https://github.com/mcrute)) +- Added `service.beta.kubernetes.io/aws-load-balancer-security-groups` annotation to set the security groups to the AWS ELB to be the only ones specified in the annotation in case this is present (does not add `0.0.0.0/0`). ([#62774](https://github.com/kubernetes/kubernetes/pull/62774), [@Raffo](https://github.com/Raffo)) + +### SIG Azure + +- Ensured orphan public IPs on Azure deleted when service recreated with the same name. ([#70463](https://github.com/kubernetes/kubernetes/pull/70463), [@feiskyer](https://github.com/feiskyer)) +- Improved Azure instance metadata handling by adding caches. ([#70353](https://github.com/kubernetes/kubernetes/pull/70353), [@feiskyer](https://github.com/feiskyer)) +- Corrected check for non-Azure managed nodes with the Azure cloud provider ([#70135](https://github.com/kubernetes/kubernetes/pull/70135), [@marc-sensenich](https://github.com/marc-sensenich)) +- Fixed azure disk attach/detach failed forever issue ([#71377](https://github.com/kubernetes/kubernetes/pull/71377), [@andyzhangx](https://github.com/andyzhangx)) +- DisksAreAttached --> getNodeDataDisks--> GetDataDisks --> getVirtualMachine --> vmCache.Get ([#71495](https://github.com/kubernetes/kubernetes/pull/71495), [@andyzhangx](https://github.com/andyzhangx)) + +### SIG CLI + +- `kubectl apply` can now change a deployment strategy from rollout to recreate without explicitly clearing the rollout-related fields ([#70436](https://github.com/kubernetes/kubernetes/pull/70436), [@liggitt](https://github.com/liggitt)) +- The `kubectl plugin list` command now displays discovered plugin paths in the same order as they are found in a user's PATH variable. ([#70443](https://github.com/kubernetes/kubernetes/pull/70443), [@juanvallejo](https://github.com/juanvallejo)) +- `kubectl get` no longer exits before printing all of its results if an error is found ([#70311](https://github.com/kubernetes/kubernetes/pull/70311), [@juanvallejo](https://github.com/juanvallejo)) +- Fixed a runtime error occuring when sorting the output of `kubectl get` with empty results ([#70740](https://github.com/kubernetes/kubernetes/pull/70740), [@mfpierre](https://github.com/mfpierre)) +- kubectl: support multiple arguments for cordon/uncordon and drain ([#68655](https://github.com/kubernetes/kubernetes/pull/68655), [@goodluckbot](https://github.com/goodluckbot)) +- Fixed ability for admin/edit/view users to see controller revisions, needed for kubectl rollout commands ([#70699](https://github.com/kubernetes/kubernetes/pull/70699), [@liggitt](https://github.com/liggitt)) +- `kubectl rollout undo` now returns errors when attempting to rollback a deployment to a non-existent revision ([#70039](https://github.com/kubernetes/kubernetes/pull/70039), [@liggitt](https://github.com/liggitt)) +- kubectl run now generates apps/v1 deployments by default ([#71006](https://github.com/kubernetes/kubernetes/pull/71006), [@liggitt](https://github.com/liggitt)) +- The "kubectl cp" command now supports path shortcuts (../) in remote paths. ([#65189](https://github.com/kubernetes/kubernetes/pull/65189), [@juanvallejo](https://github.com/juanvallejo)) +- Fixed dry-run output in kubectl apply --prune ([#69344](https://github.com/kubernetes/kubernetes/pull/69344), [@zegl](https://github.com/zegl)) +- The kubectl wait command must handle when a watch returns an error vs closing by printing out the error and retrying the watch. ([#69389](https://github.com/kubernetes/kubernetes/pull/69389), [@smarterclayton](https://github.com/smarterclayton)) +- kubectl: support multiple arguments for cordon/uncordon and drain ([#68655](https://github.com/kubernetes/kubernetes/pull/68655), [@goodluckbot](https://github.com/goodluckbot)) + +### SIG Cloud Provider + +- Added deprecation warning for all cloud providers ([#69171](https://github.com/kubernetes/kubernetes/pull/69171), [@andrewsykim](https://github.com/andrewsykim)) + +### SIG Cluster Lifecycle + +- kubeadm: Updates version of CoreDNS to 1.2.6 ([#70796](https://github.com/kubernetes/kubernetes/pull/70796), [@detiber](https://github.com/detiber)) +- kubeadm: Validate kubeconfig files in case of external CA mode. ([#70537](https://github.com/kubernetes/kubernetes/pull/70537), [@yagonobre](https://github.com/yagonobre)) +- kubeadm: The writable config file option for extra volumes is renamed to readOnly with a reversed meaning. With readOnly defaulted to false (as in pod specs). ([#70495](https://github.com/kubernetes/kubernetes/pull/70495), [@rosti](https://github.com/rosti)) +- kubeadm: Multiple API server endpoints support upon join is removed as it is now redundant. ([#69812](https://github.com/kubernetes/kubernetes/pull/69812), [@rosti](https://github.com/rosti)) +- `kubeadm reset` now cleans up custom etcd data path ([#70003](https://github.com/kubernetes/kubernetes/pull/70003), [@yagonobre](https://github.com/yagonobre)) +- kubeadm: Fixed unnecessary upgrades caused by undefined order of Volumes and VolumeMounts in manifests ([#70027](https://github.com/kubernetes/kubernetes/pull/70027), [@bart0sh](https://github.com/bart0sh)) +- kubeadm: Fixed node join taints. ([#69846](https://github.com/kubernetes/kubernetes/pull/69846), [@andrewrynhard](https://github.com/andrewrynhard)) +- Fixed cluster autoscaler addon permissions so it can access batch/job. ([#69858](https://github.com/kubernetes/kubernetes/pull/69858), [@losipiuk](https://github.com/losipiuk)) +- kubeadm: JoinConfiguration now houses the discovery options in a nested Discovery structure, which in turn has a couple of other nested structures to house more specific options (BootstrapTokenDiscovery and FileDiscovery) ([#67763](https://github.com/kubernetes/kubernetes/pull/67763), [@rosti](https://github.com/rosti)) +- kubeadm: Fixed a possible scenario where kubeadm can pull much newer control-plane images ([#69301](https://github.com/kubernetes/kubernetes/pull/69301), [@neolit123](https://github.com/neolit123)) +- kubeadm now allows mixing of init/cluster and join configuration in a single YAML file (although a warning gets printed in this case). ([#69426](https://github.com/kubernetes/kubernetes/pull/69426), [@rosti](https://github.com/rosti)) +- kubeadm: Added a `v1beta1` API. ([#69289](https://github.com/kubernetes/kubernetes/pull/69289), [@fabriziopandini](https://github.com/fabriziopandini)) +- kubeadm init correctly uses `--node-name` and `--cri-socket` when `--config` option is also used ([#71323](https://github.com/kubernetes/kubernetes/pull/71323), [@bart0sh](https://github.com/bart0sh)) +- kubeadm: Always pass spec.nodeName as `--hostname-override` for kube-proxy ([#71283](https://github.com/kubernetes/kubernetes/pull/71283), [@Klaven](https://github.com/Klaven)) +- `kubeadm join` correctly uses `--node-name` and `--cri-socket` when `--config` option is also used ([#71270](https://github.com/kubernetes/kubernetes/pull/71270), [@bart0sh](https://github.com/bart0sh)) +- kubeadm now supports the `--image-repository` flag for customizing what registry to pull images from ([#71135](https://github.com/kubernetes/kubernetes/pull/71135), [@luxas](https://github.com/luxas)) +- kubeadm: The writable config file option for extra volumes is renamed to readOnly with a reversed meaning. With readOnly defaulted to false (as in pod specs). ([#70495](https://github.com/kubernetes/kubernetes/pull/70495), [@rosti](https://github.com/rosti)) +- kubeadm: Multiple API server endpoints support upon join is removed as it is now redundant. ([#69812](https://github.com/kubernetes/kubernetes/pull/69812), [@rosti](https://github.com/rosti)) +- kubeadm: JoinConfiguration now houses the discovery options in a nested Discovery structure, which in turn has a couple of other nested structures to house more specific options (BootstrapTokenDiscovery and FileDiscovery) ([#67763](https://github.com/kubernetes/kubernetes/pull/67763), [@rosti](https://github.com/rosti)) +- kubeadm: Added a `v1beta1` API. ([#69289](https://github.com/kubernetes/kubernetes/pull/69289), [@fabriziopandini](https://github.com/fabriziopandini)) +- kubeadm: Use `advertise-client-urls` instead of `listen-client-urls` as and `etcd-servers` options for apiserver. ([#69827](https://github.com/kubernetes/kubernetes/pull/69827), [@tomkukral](https://github.com/tomkukral)) +- Kubeadm now respects the custom image registry configuration across joins and upgrades. Kubeadm passes the custom registry to the kubelet for a custom pause container. ([#70603](https://github.com/kubernetes/kubernetes/pull/70603), [@chuckha](https://github.com/chuckha)) +- `kubeadm reset` now outputs instructions about manual iptables rules cleanup. ([#70874](https://github.com/kubernetes/kubernetes/pull/70874), [@rdodev](https://github.com/rdodev)) +- kubeadm: remove the AuditPolicyConfiguration feature gate ([#70807](https://github.com/kubernetes/kubernetes/pull/70807), [@Klaven](https://github.com/Klaven)) +- kubeadm pre-pulls Etcd image only if external Etcd is not used and ([#70743](https://github.com/kubernetes/kubernetes/pull/70743), [@bart0sh](https://github.com/bart0sh)) +- kubeadm: UnifiedControlPlaneImage is replaced by UseHyperKubeImage boolean value. ([#70793](https://github.com/kubernetes/kubernetes/pull/70793), [@rosti](https://github.com/rosti)) +- For kube-up and derived configurations, CoreDNS will honor master taints, for consistency with kube-dns behavior. ([#70868](https://github.com/kubernetes/kubernetes/pull/70868), [@justinsb](https://github.com/justinsb)) +- Recognize newer docker versions without -ce/-ee suffix: 18.09.0 ([#71001](https://github.com/kubernetes/kubernetes/pull/71001), [@thomas-riccardi](https://github.com/thomas-riccardi)) +- Any external provider should be aware the cloud-provider interface should be imported from :- ([#68310](https://github.com/kubernetes/kubernetes/pull/68310), [@cheftako](https://github.com/cheftako)) +- Fixed 'kubeadm upgrade' infinite loop waiting for pod restart ([#69886](https://github.com/kubernetes/kubernetes/pull/69886), [@bart0sh](https://github.com/bart0sh)) +- Bumped addon-manager to v8.8 ([#69337](https://github.com/kubernetes/kubernetes/pull/69337), [@MrHohn](https://github.com/MrHohn)) +- GCE: Filter out spammy audit logs from cluster autoscaler. ([#70696](https://github.com/kubernetes/kubernetes/pull/70696), [@loburm](https://github.com/loburm)) +- GCE: Enable by default audit logging truncating backend. ([#68288](https://github.com/kubernetes/kubernetes/pull/68288), [@loburm](https://github.com/loburm)) +- Bumped cluster-proportional-autoscaler to 1.3.0 ([#69338](https://github.com/kubernetes/kubernetes/pull/69338), [@MrHohn](https://github.com/MrHohn)) +- Updated defaultbackend to v1.5 ([#69334](https://github.com/kubernetes/kubernetes/pull/69334), [@bowei](https://github.com/bowei)) + +### SIG GCP + +- Added tolerations for Stackdriver Logging and Metadata Agents. ([#69737](https://github.com/kubernetes/kubernetes/pull/69737), [@qingling128](https://github.com/qingling128)) +- Enabled insertId generation, and updated Stackdriver Logging Agent image to 0.5-1.5.36-1-k8s. This help reduce log duplication and guarantee log order. ([#68920](https://github.com/kubernetes/kubernetes/pull/68920), [@qingling128](https://github.com/qingling128)) +- Updated crictl to v1.12.0 ([#69033](https://github.com/kubernetes/kubernetes/pull/69033), [@feiskyer](https://github.com/feiskyer)) + +### SIG Network + +- Corrected family type (inet6) for ipsets in ipv6-only clusters ([#68436](https://github.com/kubernetes/kubernetes/pull/68436), [@uablrek](https://github.com/uablrek)) +- kube-proxy argument `hostname-override` can be used to override hostname defined in the configuration file ([#69340](https://github.com/kubernetes/kubernetes/pull/69340), [@stevesloka](https://github.com/stevesloka)) +- CoreDNS correctly implements DNS spec for Services with externalNames that look like IP addresses. Kube-dns does not follow the spec for the same case, resulting in a behavior change when moving from Kube-dns to CoreDNS. See: [coredns/coredns#2324](https://github.com/coredns/coredns/issues/2324) +- IPVS proxier now set net/ipv4/vs/conn_reuse_mode to 0 by default, which will highly improve IPVS proxier performance. ([#71114](https://github.com/kubernetes/kubernetes/pull/71114), [@Lion-Wei](https://github.com/Lion-Wei)) +- CoreDNS is now version 1.2.6 ([#70799](https://github.com/kubernetes/kubernetes/pull/70799), [@rajansandeep](https://github.com/rajansandeep)) +- Addon configuration is introduced in the kubeadm config API, while feature flag CoreDNS is now deprecated. ([#70024](https://github.com/kubernetes/kubernetes/pull/70024), [@fabriziopandini](https://github.com/fabriziopandini)) + +### SIG Node + +- Fixed a bug in previous releases where a pod could be placed inside another pod's cgroup when specifying --cgroup-root ([#70678](https://github.com/kubernetes/kubernetes/pull/70678), [@dashpole](https://github.com/dashpole)) +- Optimized calculating stats when only CPU and Memory stats are returned from Kubelet stats/summary http endpoint. ([#68841](https://github.com/kubernetes/kubernetes/pull/68841), [@krzysztof-jastrzebski](https://github.com/krzysztof-jastrzebski)) +- kubelet now supports `log-file` option to write logs directly to a specific file ([#70917](https://github.com/kubernetes/kubernetes/pull/70917), [@dims](https://github.com/dims)) +- Do not detach volume if mount in progress ([#71145](https://github.com/kubernetes/kubernetes/pull/71145), [@gnufied](https://github.com/gnufied)) +- The runtimeHandler field on the RuntimeClass resource now accepts the empty string. ([#69550](https://github.com/kubernetes/kubernetes/pull/69550), [@tallclair](https://github.com/tallclair)) +- kube-apiserver: fixes `procMount` field incorrectly being marked as required in openapi schema ([#69694](https://github.com/kubernetes/kubernetes/pull/69694), [@jessfraz](https://github.com/jessfraz)) + +### SIG OpenStack + +- Fixed cloud-controller-manager crash when using OpenStack provider and PersistentVolume initializing controller ([#70459](https://github.com/kubernetes/kubernetes/pull/70459), [@mvladev](https://github.com/mvladev)) + +### SIG Release + +- Use debian-base instead of busybox as base image for server images ([#70245](https://github.com/kubernetes/kubernetes/pull/70245), [@ixdy](https://github.com/ixdy)) +- Images for cloud-controller-manager, kube-apiserver, kube-controller-manager, and kube-scheduler now contain a minimal /etc/nsswitch.conf and should respect /etc/hosts for lookups ([#69238](https://github.com/kubernetes/kubernetes/pull/69238), [@BenTheElder](https://github.com/BenTheElder)) + +### SIG Scheduling + +- Added metrics for volume scheduling operations ([#59529](https://github.com/kubernetes/kubernetes/pull/59529), [@wackxu](https://github.com/wackxu)) +- Improved memory use and performance when processing large numbers of pods containing tolerations ([#65350](https://github.com/kubernetes/kubernetes/pull/65350), [@liggitt](https://github.com/liggitt)) +- Fixed a bug in the scheduler that could cause the scheduler to go to an infinite loop when all nodes in a zone are removed. ([#69758](https://github.com/kubernetes/kubernetes/pull/69758), [@bsalamat](https://github.com/bsalamat)) +- Clear pod binding cache on bind error to make sure stale pod binding cache will not be used. ([#71212](https://github.com/kubernetes/kubernetes/pull/71212), [@cofyc](https://github.com/cofyc)) +- Fixed a scheduler panic due to internal cache inconsistency ([#71063](https://github.com/kubernetes/kubernetes/pull/71063), [@Huang-Wei](https://github.com/Huang-Wei)) +- Report kube-scheduler unhealthy if leader election is deadlocked. ([#71085](https://github.com/kubernetes/kubernetes/pull/71085), [@bsalamat](https://github.com/bsalamat)) +- Fixed a potential bug that scheduler preempts unnecessary pods. ([#70898](https://github.com/kubernetes/kubernetes/pull/70898), [@Huang-Wei](https://github.com/Huang-Wei)) + +### SIG Storage + +- Fixed CSI volume limits not showing up in node's capacity and allocatable ([#70540](https://github.com/kubernetes/kubernetes/pull/70540), [@gnufied](https://github.com/gnufied)) +- CSI drivers now have access to mountOptions defined on the storage class when attaching volumes. ([#67898](https://github.com/kubernetes/kubernetes/pull/67898), [@bswartz](https://github.com/bswartz)) +- change default azure file mount permission to 0777 ([#69854](https://github.com/kubernetes/kubernetes/pull/69854), [@andyzhangx](https://github.com/andyzhangx)) +- Fixed subpath in containerized kubelet. ([#69565](https://github.com/kubernetes/kubernetes/pull/69565), [@jsafrane](https://github.com/jsafrane)) +- Fixed panic on iSCSI volume tear down. ([#69140](https://github.com/kubernetes/kubernetes/pull/69140), [@jsafrane](https://github.com/jsafrane)) +- CSIPersistentVolume feature, i.e. PersistentVolumes with CSIPersistentVolumeSource, is GA. ([#69929](https://github.com/kubernetes/kubernetes/pull/69929), [@jsafrane](https://github.com/jsafrane)) +- Fixed CSIDriver API object to allow missing fields. ([#69331](https://github.com/kubernetes/kubernetes/pull/69331), [@jsafrane](https://github.com/jsafrane)) +- Flex volume plugins now support expandvolume (to increase underlying volume capacity) and expanfs (resize filesystem) commands that Flex plugin authors can implement to support expanding in use Flex PersistentVolumes ([#67851](https://github.com/kubernetes/kubernetes/pull/67851), [@aniket-s-kulkarni](https://github.com/aniket-s-kulkarni)) +- Enabled AttachVolumeLimit feature ([#69225](https://github.com/kubernetes/kubernetes/pull/69225), [@gnufied](https://github.com/gnufied)) +- The default storage class annotation for the storage addons has been changed to use the GA variant ([#68345](https://github.com/kubernetes/kubernetes/pull/68345), [@smelchior](https://github.com/smelchior)) +- GlusterFS PersistentVolumes sources can now reference endpoints in any namespace using the `spec.glusterfs.endpointsNamespace` field. Ensure all kubelets are upgraded to 1.13+ before using this capability. ([#60195](https://github.com/kubernetes/kubernetes/pull/60195), [@humblec](https://github.com/humblec)) +- Fixed GetVolumeLimits log flushing issue ([#69558](https://github.com/kubernetes/kubernetes/pull/69558), [@andyzhangx](https://github.com/andyzhangx)) +- The `MountPropagation` feature is unconditionally enabled in v1.13, and can no longer be disabled. ([#68230](https://github.com/kubernetes/kubernetes/pull/68230), [@bertinatto](https://github.com/bertinatto)) + +### SIG Windows + +- `kubelet --system-reserved` and `--kube-reserved` are supported now on Windows nodes ([#69960](https://github.com/kubernetes/kubernetes/pull/69960), [@feiskyer](https://github.com/feiskyer)) +- Windows runtime endpoints is now switched to `npipe:////./pipe/dockershim` from `tcp://localhost:3735`. ([#69516](https://github.com/kubernetes/kubernetes/pull/69516), [@feiskyer](https://github.com/feiskyer)) +- Fixed service issues with named targetPort for Windows ([#70076](https://github.com/kubernetes/kubernetes/pull/70076), [@feiskyer](https://github.com/feiskyer)) +- Handle Windows named pipes in host mounts. ([#69484](https://github.com/kubernetes/kubernetes/pull/69484), [@ddebroy](https://github.com/ddebroy)) +- Fixed inconsistency in windows kernel proxy when updating HNS policy. ([#68923](https://github.com/kubernetes/kubernetes/pull/68923), [@delulu](https://github.com/delulu)) + +## External Dependencies + +- Default etcd server is unchanged at v3.2.24 since Kubernetes 1.12. ([#68318](https://github.com/kubernetes/kubernetes/pull/68318)) +- The list of validated docker versions remain unchanged at 1.11.1, 1.12.1, 1.13.1, 17.03, 17.06, 17.09, 18.06 since Kubernetes 1.12. ([#68495](https://github.com/kubernetes/kubernetes/pull/68495)) +- The default Go version was updated to 1.11.2. ([#70665](https://github.com/kubernetes/kubernetes/pull/70665)) +- The minimum supported Go version was updated to 1.11.2 ([#69386](https://github.com/kubernetes/kubernetes/pull/69386)) +- CNI is unchanged at v0.6.0 since Kubernetes 1.10 ([#51250](https://github.com/kubernetes/kubernetes/pull/51250)) +- CSI is updated to 1.0.0. Pre-1.0.0 API support is now deprecated. ([#71020](https://github.com/kubernetes/kubernetes/pull/71020)]) +- The dashboard add-on has been updated to v1.10.0. ([#68450](https://github.com/kubernetes/kubernetes/pull/68450)) +- Heapster remains at v1.6.0-beta, but is now retired in Kubernetes 1.13 ([#67074](https://github.com/kubernetes/kubernetes/pull/67074)) +- Cluster Autoscaler has been upgraded to v1.13.0 ([#71513](https://github.com/kubernetes/kubernetes/pull/71513)) +- kube-dns is unchanged at v1.14.13 since Kubernetes 1.12 ([#68900](https://github.com/kubernetes/kubernetes/pull/68900)) +- Influxdb is unchanged at v1.3.3 since Kubernetes 1.10 ([#53319](https://github.com/kubernetes/kubernetes/pull/53319)) +- Grafana is unchanged at v4.4.3 since Kubernetes 1.10 ([#53319](https://github.com/kubernetes/kubernetes/pull/53319)) +- Kibana has been upgraded to v6.3.2. ([#67582](https://github.com/kubernetes/kubernetes/pull/67582)) +- CAdvisor has been updated to v0.32.0 ([#70964](https://github.com/kubernetes/kubernetes/pull/70964)) +- fluentd-gcp-scaler has been updated to v0.5.0 ([#68837](https://github.com/kubernetes/kubernetes/pull/68837)) +- Fluentd in fluentd-elasticsearch is unchanged at v1.2.4 since Kubernetes 1.11 ([#67434](https://github.com/kubernetes/kubernetes/pull/67434)) +- fluentd-elasticsearch has been updated to v2.2.1 ([#68012](https://github.com/kubernetes/kubernetes/pull/68012)) +- The fluent-plugin-kubernetes_metadata_filter plugin in fluentd-elasticsearch is unchanged at 2.0.0 since Kubernetes 1.12 ([#67544](https://github.com/kubernetes/kubernetes/pull/67544)) +- fluentd-gcp has been updated to v3.2.0 ([#70954](https://github.com/kubernetes/kubernetes/pull/70954)) +- OIDC authentication is unchanged at coreos/go-oidc v2 since Kubernetes 1.10 ([#58544](https://github.com/kubernetes/kubernetes/pull/58544)) +- Calico was updated to v3.3.1 ([#70932](https://github.com/kubernetes/kubernetes/pull/70932)) +- Upgraded crictl on GCE to v1.12.0 ([#69033](https://github.com/kubernetes/kubernetes/pull/69033)) +- CoreDNS has been updated to v1.2.6 ([#70799](https://github.com/kubernetes/kubernetes/pull/70799)) +- event-exporter has been updated to v0.2.3 ([#67691](https://github.com/kubernetes/kubernetes/pull/67691)) +- Es-image remains unchanged at Elasticsearch 6.3.2 since Kubernetes 1.12 ([#67484](https://github.com/kubernetes/kubernetes/pull/67484)) +- metrics-server remains unchanged at v0.3.1 since Kubernetes 1.12 ([#68746](https://github.com/kubernetes/kubernetes/pull/68746)) +- GLBC remains unchanged at v1.2.3 since Kubernetes 1.12 ([#66793](https://github.com/kubernetes/kubernetes/pull/66793)) +- Ingress-gce remains unchanged at v1.2.3 since Kubernetes 1.12 ([#66793](https://github.com/kubernetes/kubernetes/pull/66793)) +- ip-masq-agen remains unchanged at v2.1.1 since Kubernetes 1.12 ([#67916](https://github.com/kubernetes/kubernetes/pull/67916)) +- [v1.13.0-rc.2](#v1130-rc2) +- [v1.13.0-rc.1](#v1130-rc1) +- [v1.13.0-beta.2](#v1130-beta2) +- [v1.13.0-beta.1](#v1130-beta1) +- [v1.13.0-alpha.3](#v1130-alpha3) +- [v1.13.0-alpha.2](#v1130-alpha2) +- [v1.13.0-alpha.1](#v1130-alpha1) + + + +# v1.13.0-rc.2 + +[Documentation](https://docs.k8s.io) + +## Downloads for v1.13.0-rc.2 + + +filename | sha512 hash +-------- | ----------- +[kubernetes.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes.tar.gz) | `12fbaf943ae72711cd93c9955719ec1773a229dbb8f86a44fcda179229beb82add4dc1a54ceb50b9f48fde48e2464ed0cd4b2e57d9689a7ae784cb052beb6751` +[kubernetes-src.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-src.tar.gz) | `8e94f0fe73909610e85c201bb1ba4f66fd55ca2b4ded77217a4dfad2874d402cc1cc94203ecc195f909126c186701e5e1e62890ad288895493a1759f88a190d0` + +### Client Binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-client-darwin-386.tar.gz) | `ac555f5d1e6b88fa4de1e06e0a1ebd372582f97c526c938334a8c63fbf17545607efbba9975d1767e147113e551e986d6523f6985ea41236cfbf7949df31f016` +[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-client-darwin-amd64.tar.gz) | `2eae428a0e4bcb2237343d7ac1e431ccfc1f7037622bb3131ad8d48a3af6f5ed34be899ec1ec32af7eb7d411cb0cda02a2413405479722ab868cdc816726c9df` +[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-client-linux-386.tar.gz) | `89e671679b4516f184f7fd5ea0fe2a9ab0245fab34447625786bf55841223124527d3aa2ee6fa2474333f37eea4e9a5ba6f3f4dc3698907fd24bedf522f53b40` +[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-client-linux-amd64.tar.gz) | `61f6513722e9c485300b822d6fc5998927bbffa18862d2d3f177a7c7cc0ee56c51ec169e3c8239e352c022094bb02124ed060d7d5c3cec9b67aae20ffd42f387` +[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-client-linux-arm.tar.gz) | `ef0e5fd4bf2074dfd3cf54d45307550273695906baca3533a9d23424e7b693d706f6d1d3a09a34e2d1f84d9eddc6b62d96e5190b8c7145919e93f0ae75ec4d06` +[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-client-linux-arm64.tar.gz) | `d34bb9ce9bfe2a5375fd58920e63b4eef818348719dba460f35838433af57a1a23fa659e53de52c8174fa212c94c4196ac5a02ce02ef714860488c77563b5821` +[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-client-linux-ppc64le.tar.gz) | `4dc4e4a5e166e63360ba86e1278bbe75212ac7c3f60ba30425a1c5654bf5a9b1164543fdc23d7dfd9d3aea7be38544c8dc535459e96c062db631e58c5c628762` +[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-client-linux-s390x.tar.gz) | `d27675f4753469cd5e31faed13a1ea9654c25d38b0d96c1340215fd231050ffc66dc40c5103f8377339bacf00f1c99d386fe9c21fc68c5a21c10667f773d9d4b` +[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-client-windows-386.tar.gz) | `9d6e6de2d4a55eaeebd7fa6b861548e0768381d50838430722b56636428a3417b8f2bbc953bc365294a857d8f5b51d90807e5eafe874f37d9b726f48b5d04197` +[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-client-windows-amd64.tar.gz) | `30b2da5c015ef88b9efcf90bffe0498d367df7c126b65f2e878af263c5d62b8c93792dbf20511d0ff034c7a9e2c3fc93931860e1254ed158eddec34f407b9005` + +### Server Binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-server-linux-amd64.tar.gz) | `8180f2b788249fe65f7f1d3ee431ac758ede29a6349db312afbee080ff2c24586fc468f11a9cbcb8d22842739974f29e10793778f5fd5c55d10129e97a1efce3` +[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-server-linux-arm.tar.gz) | `e9165284a0b82a9ab88dad05f43bfe1bebecad3bb1c7118475c3426e0b6f9f91d340e1e6223d81df9337ab4cc9a96708443c025030127acf88437f0c327b750b` +[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-server-linux-arm64.tar.gz) | `03797c021ebed3b08835e72eed405c57aaacce972bbbbf88bf49310efbf8c7242f2f223d73b5d2ed4c21e5196e6e5fb7b2b811f08607db6dbe98f869bf28bedb` +[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-server-linux-ppc64le.tar.gz) | `ceb49af22e3b518f3ba27c1e7de28e577e2735175e84a6d203f1f8766eceaa7c0424746ff71498d7847e98f538af5663b16cc306cb0adbb006d5d869766dfb9b` +[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-server-linux-s390x.tar.gz) | `bee4752e8a52e217ae1ffcfbc263453c724de684b4d463d5ddb24a3a30a67fc8f78e6c0a8154c6b6581d17f1e168903bc18d0e56f02fce5933f673bb4c74a8cf` + +### Node Binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-node-linux-amd64.tar.gz) | `b368989bbb8ab4d29b51d5d4d71d073b0ceb39614c944859dcd14c3303c31475850f7012deaa8d5ba9c17edd728bce536fbd523ae7defc74a30f0878f05497bf` +[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-node-linux-arm.tar.gz) | `404b7b74a1e0d0fed9088a7e9461e02cfd9a6992c554baa125b7a361a6baa03d1e4622fbc4ec51836f00a7ac4f90167f345307678527f5781e06acdf526b9a45` +[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-node-linux-arm64.tar.gz) | `fa531b1675a778c572a2175fb1bed00e78dc589f638f2096b3b5c9d3d691a5668787a43d69898678abd70c7b949e05cfebfb0783c0144a66bdff61fed6094582` +[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-node-linux-ppc64le.tar.gz) | `a7ecc1f63e632c1b4f9b312babd6882ec966420bf4f8346edf80495fcf860d912729072c79d23cc071a07239783409b02c1f4a716a24e2597f2b490c9b3bb5b3` +[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-node-linux-s390x.tar.gz) | `a7171ed95de943a0ac5a32da4458e8d4366eb1fadbe426bebc371d2bb6536636b14db9d2cd03952258b3cb1b99fdca2db07947b028cc6c7bb92f4281ba6f62f2` +[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.13.0-rc.2/kubernetes-node-windows-amd64.tar.gz) | `8a3a71d142b99fb200c4c1c9c0fa4dc6a3b64a0b506dc37dc3d832a94a791619a09ae4b2c6f73802f6833234570633974547f7700c8bb6de71d91ba2c4ac4b54` + +## Changelog since v1.13.0-rc.1 ### Other notable changes -* Update to use manifest list for etcd image ([#68896](https://github.com/kubernetes/kubernetes/pull/68896), [@ixdy](https://github.com/ixdy)) -* Fix Azure nodes power state for InstanceShutdownByProviderID() ([#68921](https://github.com/kubernetes/kubernetes/pull/68921), [@feiskyer](https://github.com/feiskyer)) -* Bump kube-dns to 1.14.13 ([#68900](https://github.com/kubernetes/kubernetes/pull/68900), [@MrHohn](https://github.com/MrHohn)) - * - Update Alpine base image to 3.8.1. - * - Build multi-arch images correctly. -* kubelet: fix grpc timeout in the CRI client ([#67793](https://github.com/kubernetes/kubernetes/pull/67793), [@fisherxu](https://github.com/fisherxu)) -* Update to golang 1.10.4 ([#68802](https://github.com/kubernetes/kubernetes/pull/68802), [@ixdy](https://github.com/ixdy)) -* kubeadm now uses fat manifests for the kube-dns images ([#68830](https://github.com/kubernetes/kubernetes/pull/68830), [@rosti](https://github.com/rosti)) -* Update Cluster Autoscaler version to 1.12.0. ([#68739](https://github.com/kubernetes/kubernetes/pull/68739), [@losipiuk](https://github.com/losipiuk)) - * See https://github.com/kubernetes/autoscaler/releases/tag/1.12.0 for CA release notes. -* kube-proxy restores the *filter table when running in ipvs mode. ([#68786](https://github.com/kubernetes/kubernetes/pull/68786), [@alexjx](https://github.com/alexjx)) -* New kubeDNS image fixes an issue where SRV records were incorrectly being compressed. Added manifest file for multiple arch images. ([#68430](https://github.com/kubernetes/kubernetes/pull/68430), [@prameshj](https://github.com/prameshj)) -* Drain should delete terminal pods. ([#68767](https://github.com/kubernetes/kubernetes/pull/68767), [@ravisantoshgudimetla](https://github.com/ravisantoshgudimetla)) +* Update Cluster Autoscaler version to 1.13.0. Release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.13.0 ([#71513](https://github.com/kubernetes/kubernetes/pull/71513), [@losipiuk](https://github.com/losipiuk)) +* fix detach azure disk issue due to dirty cache ([#71495](https://github.com/kubernetes/kubernetes/pull/71495), [@andyzhangx](https://github.com/andyzhangx)) -# v1.12.0-rc.1 +# v1.13.0-rc.1 -[Documentation](https://docs.k8s.io) & [Examples](https://releases.k8s.io/release-1.12/examples) +[Documentation](https://docs.k8s.io) -## Downloads for v1.12.0-rc.1 +## Downloads for v1.13.0-rc.1 -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes.tar.gz) | `ac65cf9571c3a03105f373db23c8d7f4d01fe1c9ee09b06615bb02d0b81d572c` -[kubernetes-src.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-src.tar.gz) | `28518e1d9c7fe5c54aa3b57235ac8d1a7dae02aec04177c38ca157fc2d16edb6` +[kubernetes.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes.tar.gz) | `1c047e4edcf3553a568679e6e5083988b06df9d938f299a9193c72ad96a9c439a1f47f98b86f75d94746e8c1ae363b7a3de3c29dbdf7585b5e5e67b95f309d4a` +[kubernetes-src.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-src.tar.gz) | `d2fd47c38abd29a2037b9e2a3a958ec250e2c6ae77532f6e935a6422bd626485fd720932b18fe2fdfcc7b17c6014a9da08cd9e6f9272f19f666ec52ffc02b564` ### Client Binaries -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-client-darwin-386.tar.gz) | `7b6f6f264464d40b7975baecdd796d4f75c5a305999b4ae1f4513646184cac7c` -[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-client-darwin-amd64.tar.gz) | `5feabe3e616125a36ce4c8021d6bdccdec0f3d82f151b80af7cac1453255b4d5` -[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-client-linux-386.tar.gz) | `40524a1a09dd24081b3494593a02a461227727f8706077542f2b8603e1cf7e06` -[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-client-linux-amd64.tar.gz) | `ac2c9757d7df761bdf8ffc259fff07448c300dd110c7dbe2ae3830197eb023e9` -[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-client-linux-arm.tar.gz) | `02f27ae16e8ebb12b3cb66391fe85f64de08a99450d726e9defd2c5bcd590955` -[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-client-linux-arm64.tar.gz) | `1286af2cad3f8e2ee8e2dc18a738935779631b58e7ef3da8794bbeadca2f332e` -[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-client-linux-ppc64le.tar.gz) | `9c04419b159fb0fe501d6e0c8122d6a80b5d6961070ebc5e759f4327a1156cf4` -[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-client-linux-s390x.tar.gz) | `104d5c695826971c64cb0cec26cf791d609d3e831edb33574e9af2c4b191f049` -[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-client-windows-386.tar.gz) | `0096f8126eb04eafa9decd258f6d09977d24eee91b83781347a34ebb7d2064aa` -[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-client-windows-amd64.tar.gz) | `a641a1a421795279a6213163d7becab9dc6014362e6566f13d660ef1638dc286` +[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-client-darwin-386.tar.gz) | `44d0733359be5036953775e12fc1723e4c64452a24a8c3b522c8a624e0a132cf61483a120cafebe1370939b38ddf1809969dfc0daf0c087ce8a888aa98f2fa6f` +[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-client-darwin-amd64.tar.gz) | `2acd37ed234271b0ff9c30273261e4b127309a1bc91a006b7a07e1a948703fa550699cd7f44dceb4e7cc6be139f80785853ce4dedb3c3d3f0df85598d0488d56` +[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-client-linux-386.tar.gz) | `5fe07ea2f776086df0e9447b7e6b0863c5b3af71f5aff8e302087e242d78613278023a169f211be96feab5109d801c9e4f427a911221d039e4d9cadec3086ebf` +[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-client-linux-amd64.tar.gz) | `7541d5850d74156862e5fe00817bd954d2b49b2c0cf15abe5cde34406928b8ca34b6907eea51e79e005156964ea1269102f2663e667ccbb4223ea12edfc97c20` +[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-client-linux-arm.tar.gz) | `122121d3e469b6e33cc3fd910b32a5a94b9d3479f0367c54fbc4e7f13df7b097c061b0624b36c0e59f9a35dda7d021f04d400506e6f40eff657672ee53b91a69` +[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-client-linux-arm64.tar.gz) | `5e3d415db4239f27461c4ea404903cfc762084d5c1e84f9ed8bc0325d7fa845ac540a279e3bd67ac80d00fcad4860398166f55f76ba22c1060e0bc1c867b2464` +[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-client-linux-ppc64le.tar.gz) | `8651f4161569913b616695bdd1a41c4b177cbfb4773fbca649b3e97957f6c5f46f4fa84bfa92ba24abc34b90cc9543d3c0707962d28d701ef784c764ef49f407` +[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-client-linux-s390x.tar.gz) | `920b81f6bbc7e7d4fa2f9c61fbc6f529621f2f134dbbb0f407866ffd0ec47791484187c609cca3b615034a5393869ae8f156a7bd0001d0ef59f195d8aab7229d` +[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-client-windows-386.tar.gz) | `0d49277cb7c36e5538d4c1c0fd6e6a69da7cd73c226f5869b29fad1e5b9bf434ffc8423b72d807df67b6674541a370a5235881bccff1b9f390f175064020453a` +[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-client-windows-amd64.tar.gz) | `34ae587e2d439f925d1e324d2bbff3a751bb73b18e98b13c93e5742e7e16c00b4d9956b91721b4e06a00087dc00862248e6de167683a6bd1ccd14b1a6dcef753` ### Server Binaries -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-server-linux-amd64.tar.gz) | `202958d3cfb774fd065ad1ec2477dc9c92ce7f0ff355807c9a2a3a61e8dad927` -[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-server-linux-arm.tar.gz) | `474de8f6a58d51eb01f6cc73b41897351528a839f818d5c4f828a484f8bc988b` -[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-server-linux-arm64.tar.gz) | `dbd5affd244815bf45ac0c7a56265800864db623a6a37e7ce9ebe5e5896453f8` -[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-server-linux-ppc64le.tar.gz) | `a62fefa8ad7b3fbfeb7702dac7d4d6f37823b6c3e4edae3356bf0781b48e42e1` -[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-server-linux-s390x.tar.gz) | `0f77690f87503c8ee7ccb473c9d2b9d26420292defd82249509cf50d8bb1a16c` +[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-server-linux-amd64.tar.gz) | `7030ef7463bef0871e524a5233d23c5f8aee18ac92e96555910ddc7a891772d451dac08b583f391132c654eaaea788f3bf29fb510a2f6f3b24b0ac79ca669f77` +[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-server-linux-arm.tar.gz) | `ccd1f413ad357581a904d1ff67f3e376be7882bd72efb13657f8aa1191c4481691743016a1385b777b5e62fe9854c1695ffa847c3b4534459317d0d5b5baaf76` +[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-server-linux-arm64.tar.gz) | `ff589f5b6c56713818edda8ae9b39b17dfbf34e881c09736f722de5d70e6dd1508b5fefc60f40547dfd4fddb32ddce2a4470e1d240b315db5840a0fba957d553` +[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-server-linux-ppc64le.tar.gz) | `f748985751bf403bc7b1f9160ce937cd2915552b27c3c79764a66789dc39ef9e3069e6f25d21e15bfaf81c535e3ee3b195eb636965456467ee56c0167c526129` +[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-server-linux-s390x.tar.gz) | `b3b0075948d72784defe94073dff251b79083aa46b4f29419026757665cac554356486948a41b59293904238651a733747a0e271f43a72228c6c83cf8f5634a7` ### Node Binaries -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-node-linux-amd64.tar.gz) | `2191845147d5aab08f14312867f86078b513b6aff8685bb8ce84a06b78ae9914` -[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-node-linux-arm.tar.gz) | `54de98d7d2a71b78bc7a45e70a2005144d210401663f5a9daadedd05f89291f0` -[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-node-linux-arm64.tar.gz) | `a765514e0c4865bb20ceb476af83b9d9356c9b565cfe12615ecf7ad3d5a6b4f7` -[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-node-linux-ppc64le.tar.gz) | `b7ae7d159602d0b933614071f11216ede4df3fc2b28a30d0018e06b3bb22cf6e` -[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-node-linux-s390x.tar.gz) | `7d4f502eda6aa70b7a18420344abfaec740d74a1edffcb9869e4305c22bba260` -[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.12.0-rc.1/kubernetes-node-windows-amd64.tar.gz) | `ed5516b1f66a39592a101bec135022b3905a66ae526b8ed3e2e9dff5ed68eda0` +[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-node-linux-amd64.tar.gz) | `01907a104c043607985053571183b7bdccf655f847d1dd9d8991cd2c464ddf9953f25cacb255be3067c1b65f6168fe92a90162636e6c6b6ec33340926d537959` +[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-node-linux-arm.tar.gz) | `dbf1801c456312698253767dd36b186fb4e503a03454cd16bba68a1ede9d29e14939591eb39516129bc8c88e64fba2a287ae6447b7e4ff4afcecd1fb50713403` +[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-node-linux-arm64.tar.gz) | `15f3259370f1419fcc372a28faa9a3caae5f2c89ee76286c14ea62d612fdca94ac7358a3cd76877736389080d28ba65237fc0aeffed2050bac4e342877351e51` +[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-node-linux-ppc64le.tar.gz) | `00dc7f5bd40d045baeb72d5dcfb302b8566aacc23cd7de1b877724e1160ee1608b3b121358d2c3b081d06deb1a8107d0437d3d1b20df88e4adcfd5f4a05964ee` +[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-node-linux-s390x.tar.gz) | `2b80e4dffa0b8bdc0305d1263c06320918541f3a7b6519123752b89be335a2c48965b7d16d814ffc02e304e9cf932db0c780fc316c99a080bebd880d55e3c939` +[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.13.0-rc.1/kubernetes-node-windows-amd64.tar.gz) | `600b442a1665e39621fce03ad07b162e2353cc8bc982cad849dab7e1c2db34bde675ef12a4907a75f2ba82e62ae3126e189b3d69db7fcab71548bf6940351dda` -## Changelog since v1.12.0-beta.2 +## Changelog since v1.13.0-beta.2 + +### Other notable changes + +* CVE-2018-1002105: Fix critical security issue in kube-apiserver upgrade request proxy handler ([#71411](https://github.com/kubernetes/kubernetes/issues/71411), [@liggitt](https://github.com/liggitt)) +* Update Cluster Autoscaler version to 1.13.0-rc.2. Release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.13.0-rc.2 ([#71452](https://github.com/kubernetes/kubernetes/pull/71452), [@losipiuk](https://github.com/losipiuk)) +* Upgrade Stackdriver Logging Agent addon image to 0.6-1.6.0-1 to use Fluentd v1.2. This provides nanoseconds timestamp granularity for logs. ([#70954](https://github.com/kubernetes/kubernetes/pull/70954), [@qingling128](https://github.com/qingling128)) +* fixes a runtime error occuring when sorting the output of `kubectl get` with empty results ([#70740](https://github.com/kubernetes/kubernetes/pull/70740), [@mfpierre](https://github.com/mfpierre)) +* fix azure disk attach/detach failed forever issue ([#71377](https://github.com/kubernetes/kubernetes/pull/71377), [@andyzhangx](https://github.com/andyzhangx)) +* Do not detach volume if mount in progress ([#71145](https://github.com/kubernetes/kubernetes/pull/71145), [@gnufied](https://github.com/gnufied)) + + + +# v1.13.0-beta.2 + +[Documentation](https://docs.k8s.io) + +## Downloads for v1.13.0-beta.2 + + +filename | sha512 hash +-------- | ----------- +[kubernetes.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes.tar.gz) | `e8607473e2b3946a3655fa895c2b7dee74818b4c2701047fee5343ab6b2f2aa3d97b19b11c7e7aeaca322a95bf99cbb5a7dafca187922fd40eaf24daaaf3bc8d` +[kubernetes-src.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-src.tar.gz) | `6ca15ad729a82b41587e1dbbd4e9ad5447e202e8e7ee8c01c411090031ee3feb83f0cc65e211e8634a01e7c52d5f1f7b47cd2ac601708542227853f312401e8f` + +### Client Binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-client-darwin-386.tar.gz) | `5727218280ea7c68350aa5cf04e3d3c346f97d462e3f60f5196e27358f71841e19523b277a5b8fe9cea4b8fa323c54a820ae1956937922bcb24524612435e699` +[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-client-darwin-amd64.tar.gz) | `3e3975a41da08135dc654a40acb86ce862b1f56a9361e0c38c9c99c5b5bcad970f2271ae9a17e03c3d6e13ed03176e5e80313b39a8a02680a3b6d806e33f3a1b` +[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-client-linux-386.tar.gz) | `26cfa99fbe09b20ebe3d2aebb4d08f0f9f2661d5533b94daf6c8354701b1e4ddb8981c10323073c0d06e52eeb0f68839726b684e4e2222b1be7b940ff6017c72` +[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-client-linux-amd64.tar.gz) | `42204953b02af81bb5f695c957aca9fa382609447ada5e3a9701da3e8bbd54923084e0b28dd5be455f39ec0dd5c4bf4e81704542d1ce64d7292c17f0a9b04a32` +[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-client-linux-arm.tar.gz) | `c680c94699b0b319b654a4c1c0a9b7fc387c44fb22744f30049142b17c3fabd3ba5358904cf8d5ccb077d0fb96d0360d222616980a13f91fc4a64ba2afce35b8` +[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-client-linux-arm64.tar.gz) | `aa997b3428979ba2652fd251c4c5ece87043472ebe2ee15d8a179e69ddbefd47e8030e9392c4f6659b8207fcfb45b2effb69d9736af7c6c82f4217c5e3ac89e1` +[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-client-linux-ppc64le.tar.gz) | `684dfc462d84d3902e322535997e57f7874003ab17c41508c057bc7c6220062cf57d0486086d28940d9b4c0e8b5ebead5d4a64ad6e23895ecd2f8841844a8527` +[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-client-linux-s390x.tar.gz) | `ff98b3a23dfe436a12843eb388be9568cbc29c9328648a1d166518aac40841bd8d855916918259cd92a0cc465e376ab1cb4c7b2b92c1eee454a76495772dc7e1` +[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-client-windows-386.tar.gz) | `6897a0f59fb409526dae9c86680702f3d2a1dc68d145504ed2e98b05d8f1dcc9b6a977c1af17277775b64501c86d5392e2f052aded8aec0c0640d6e78f609b87` +[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-client-windows-amd64.tar.gz) | `6ed67eecb2b79ace8d428cbd4d07ef7d52ba4e5b3b44eb59d46aff99a7a862f158573b4c2678cbdd31ba060a0acd695fa2d8a29ad0c4e22516622c23d017c5cb` + +### Server Binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-server-linux-amd64.tar.gz) | `351292b217c1c49b5c0241da11b4be0929a5d1645bec7dd05051930df8a70090b130d3ceef2482657db16dd6e4f71013075bcd727e741b497143bc6db67c134a` +[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-server-linux-arm.tar.gz) | `88f166a7b5a3f9d9c19a5b911adb6e8e4cac1a3323b83d681f13aaf7bb285b0d016b147b4168c886efeccb3d9052c7512c1f61f7a0f59e2ba324e2827f802712` +[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-server-linux-arm64.tar.gz) | `fb4868a939eca18de17e0b606d1ab127712e277e01c02ffa96138a53973cd583bfc28cf9c2967906896740665fdb02ed53f01ef4341cad9718a8770779d99431` +[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-server-linux-ppc64le.tar.gz) | `47a4e8e96c1e8a8cc37eabd19194b9d174fa93c3feaf1384895f89c5c6836511eb9f4ff3c91dd84c05398b7d0ce2d49fd5b43ddb518b7b9ed706f224a48b18fa` +[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-server-linux-s390x.tar.gz) | `4e0823d1da55a71f001fcb07511a7b3416641ea93bfbd56b1e1e435c0a78bafbcecc873ba43808985b66b01464cbf9b60e9ad057ec78fea5080de8a5f7684bab` + +### Node Binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-node-linux-amd64.tar.gz) | `e21964063b80f52e387cd35826f3081ad0a3b62608d182e008b8b76f572442905e4b0625839d3ff28a353f1d686f8dbd15104e6b27126e3c6a579704b2106154` +[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-node-linux-arm.tar.gz) | `cb665911af59a1cf86e5d66a4cdc134dc412e9e479dd89fa0bbbaeb8324eb87d090ffb0985e31bb12b5e063bfe8c045ac797931cfb856287b05f63b53b26a524` +[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-node-linux-arm64.tar.gz) | `c172126829aea38e2238af6b62035abad6ed08d041175b0bf99792b7c608a0b27dd7f80b5ad301843cbdfee7ed2825aee9ea52d69752759a79e1148529ad1999` +[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-node-linux-ppc64le.tar.gz) | `0367940078ea9b4d46778b8406840fd2925f612304b5fa5b675fc07d5457bea524ebaf0378691af27f97d0c0fe39fffc6ad75f6db10139f377e52d0d3888252a` +[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-node-linux-s390x.tar.gz) | `74382ed862ae099b91ce6056b85b7ee4f075fbdb4e737a8448c92e20fe3a0717047a138c23e13b0a8bda3e457f4299f412ca31768e8c87e57c9faf95b7f9adda` +[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.13.0-beta.2/kubernetes-node-windows-amd64.tar.gz) | `9164c4eae920c727965caae046e1b2daabf4822e2dee2260697b22e5208a0d8c6e7ce152a5df7852e8203d4771e6dc6bf9f63159324facb1ab81ea8f212b55fb` + +## Changelog since v1.13.0-beta.1 + +### Other notable changes + +* Fix missing flags in kube-apiserver --help. ([#70204](https://github.com/kubernetes/kubernetes/pull/70204), [@imjching](https://github.com/imjching)) +* kubeadm init correctly uses --node-name and --cri-socket when --config option is also used ([#71323](https://github.com/kubernetes/kubernetes/pull/71323), [@bart0sh](https://github.com/bart0sh)) +* API server flag `--experimental-encryption-provider-config` was renamed to `--encryption-provider-config`. The old flag is accepted with a warning but will be removed in 1.14. ([#71206](https://github.com/kubernetes/kubernetes/pull/71206), [@stlaz](https://github.com/stlaz)) +* Fix missing flags in *-controller-manager --help. ([#71298](https://github.com/kubernetes/kubernetes/pull/71298), [@stewart-yu](https://github.com/stewart-yu)) +* Clear pod binding cache on bind error to make sure stale pod binding cache will not be used. ([#71212](https://github.com/kubernetes/kubernetes/pull/71212), [@cofyc](https://github.com/cofyc)) +* kubeadm: always pass spec.nodeName as --hostname-override for kube-proxy ([#71283](https://github.com/kubernetes/kubernetes/pull/71283), [@Klaven](https://github.com/Klaven)) +* kubeadm join correctly uses --node-name and --cri-socket when --config option is also used ([#71270](https://github.com/kubernetes/kubernetes/pull/71270), [@bart0sh](https://github.com/bart0sh)) +* apiserver can be configured to reject requests that cannot be audit-logged. ([#65763](https://github.com/kubernetes/kubernetes/pull/65763), [@x13n](https://github.com/x13n)) +* Kubelet Device Plugin Registration directory changed from from `{kubelet_root_dir}/plugins/` to `{kubelet_root_dir}/plugins_registry/`. Any drivers (CSI or device plugin) that were using the old path must be updated to work with this version. ([#70494](https://github.com/kubernetes/kubernetes/pull/70494), [@RenaudWasTaken](https://github.com/RenaudWasTaken)) +* When the BoundServiceAccountTokenVolumes Alpha feature is enabled, ServiceAccount volumes now use a projected volume source and their names have the prefix "kube-api-access". ([#69848](https://github.com/kubernetes/kubernetes/pull/69848), [@mikedanese](https://github.com/mikedanese)) + + + +# v1.13.0-beta.1 + +[Documentation](https://docs.k8s.io) + +## Downloads for v1.13.0-beta.1 + + +filename | sha512 hash +-------- | ----------- +[kubernetes.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes.tar.gz) | `78245b2357a5eeafd193d28f86655327edce7bbc4da142c826eba5f5c05a624cd30b2551f63da37f38e9ca3fcf6990d77a3a068cbfc9075ff19fcdb235db9acb` +[kubernetes-src.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-src.tar.gz) | `880c5a8b16215bc58b307922474703048020b38be1d41672425cd07bdcf0626a88f04a080eac13ebb63c42214454420063289ba147d0a6cdf8dc3fe942382964` + +### Client Binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-client-darwin-386.tar.gz) | `0f804c77ef6122b4b6586a507179fe0f1a383752342b3e5575e09223fdda9731acd89631ad236969e19ba1016adb28cfdb62f5b7a2b594bd68c780a5066cf370` +[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-client-darwin-amd64.tar.gz) | `0bdbd8003bcbecb4494b4778411e7d057067e78a99a7e8e8e45a3982cbe476dab822bccdeb73b989dfc5222ab8a09554f25dfb1f3d48622a1db3a45088799e49` +[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-client-linux-386.tar.gz) | `522795df77ff8543251232863cb36fe2d501671e04a5279a112aa3ffa784de93f3f567af50fffa5367fd27793e595b3e6b8c66b0cfbc99ecadbfbb279fe9e9c1` +[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-client-linux-amd64.tar.gz) | `b6481bae237e6971f7b9cc039d3b7e62d49ddd48d52dd979432fa0318a8e3e5bf1677298ffee5c3d9af07e32c50eb94948509218302f8ba966abcc27e391b282` +[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-client-linux-arm.tar.gz) | `45b8fa2557bb742a8ce16e0a69fa64fe898509418c6f9099a24bf1ab20c7d5de61f2e79f2de46c660c53eec78c45d5606a5ae5144f1f47dca54d71d353566596` +[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-client-linux-arm64.tar.gz) | `475b823a5e2c4c6e1bc49f35fbef45d1fc6e6279f5335762bad05d0f695fd033a3a81bd90ea854d8ee2c9b41f12b22f10606a2c058ea178f0b07ddb418a69499` +[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-client-linux-ppc64le.tar.gz) | `bc289b249051e9918f8f842bb98bf4d0b8951709fe5b65c2185f04b78213ec0099973f3459ffb062342a0a01739f3919758bb2724566c5e24cb30546af68fad8` +[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-client-linux-s390x.tar.gz) | `0935e0ad23a61d570de087e72f22bc3da2a34c19bb5aea0ab342f91655b4a02ba781448c35b837e0174fcd1b5d9cfb5550a1b9271c10f67a93975088df737337` +[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-client-windows-386.tar.gz) | `4833425ff040983b841722a00edd2cfa56f85099658ae04890c4e2262931e3c74d7276f3e95a74fbdd6934f32bca41297e72209ffaa1c6192bc9b6866d78582e` +[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-client-windows-amd64.tar.gz) | `156a5328834055f7b9732c762cc917cfdbf2d2fc67dd80ba89ae7dcb9c2e7d271fc4fe2b2074f98fc308f2282138b10c86becaf6ed235de0d0d3e66a6d419c5a` + +### Server Binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-server-linux-amd64.tar.gz) | `9435be5cced10252954be579408e2a253eb51dd7b649417f1e91679bce33f6ff735f1c24687994a133155b19453fbcfff4555938134001961ece419b746ecefe` +[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-server-linux-arm.tar.gz) | `8dc3d4a0c09830831efd77fdf193ed9ccc1247bd981b4811192cac38cf5ffd04042575632d4259af933959e6e774b10ac2f9e3a79903629f36dad616dd99b5d5` +[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-server-linux-arm64.tar.gz) | `f2549f87f21ea44c5d776a706c59bb2ea61d7f2cca304850aa6ad5b09c4486b16ae1be19ab9f9166a2f9f9d2ffbef6d38676c54104d424a855ce445ca6252607` +[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-server-linux-ppc64le.tar.gz) | `0c0671eaf7cf7262c95411930311bb4610f89583431738149f0ee7f8f6a55b094322fc6717db1cdd496944c66216305ac4a5e3ea782ab1c2817e23a4bc32ff9f` +[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-server-linux-s390x.tar.gz) | `ff24909b0b044924d241d6aeac9e9b4f0696c0ca7e973d56a874b02b613a45d00c30c5709326d3ad02f0e30742c1aa412e40a061e4c38631c1a538801273e565` + +### Node Binaries + +filename | sha512 hash +-------- | ----------- +[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-node-linux-amd64.tar.gz) | `235b1c4348b5779ca71a5f63121ff6a162db02bb24b4d815ec73412afedbe0ce5f969a40569e709017fbaf8f51e952d6fbe740ab3daf5d6fc932d8fac3f74c07` +[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-node-linux-arm.tar.gz) | `cd23813419a74983bdd3a3104e20684e947ef7302dcfa1802132439b21e7621ad129621c633094e74a50c2ef57a83685f425b345898e7f501f7b0b639d8ba215` +[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-node-linux-arm64.tar.gz) | `fb6283dae828f8d9275a05c6a4ea27bc1136e8e8253b5ddac52c8254813b11e8e4373038bf145a51d9680bb14dc276f8c7852e87d553cead5811550bf21eab5a` +[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-node-linux-ppc64le.tar.gz) | `f910922d422b65f6b6a8d7762a23048991695496c0fc07c3dc4f1a81e32d250d592851a2f5b19588f7ed65281495d4187e8ab61cb4711102f2d0c8766267bf5b` +[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-node-linux-s390x.tar.gz) | `d895fed57caf038afe0087ff44d2adfdd8955d18135adad9935952702e9abf2bc07289e1a7545b30f4d9f7ab154c642b4c08d74070c66f5c917196cbb37b9692` +[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.13.0-beta.1/kubernetes-node-windows-amd64.tar.gz) | `4bc09e54935d2cb4d2bad7db06831d040cc03906d8934575932ed6eab5f0bc9ab67cdb2720c919c89d8b29388bb61f228c10167c9bc9eec46bbdc10c6381500e` + +## Changelog since v1.13.0-alpha.3 ### Action Required -* Service events are now added in azure-cloud-provider for easily identify the underground errors of Azure API. ([#68212](https://github.com/kubernetes/kubernetes/pull/68212), [@feiskyer](https://github.com/feiskyer)) - * Action required: The following clusterrole and clusterrolebinding should be applied: - - ```yaml - kind: List - apiVersion: v1 - items: - - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole - metadata: - labels: - kubernetes.io/cluster-service: "true" - name: system:azure-cloud-provider - rules: - - apiGroups: [""] - resources: ["events"] - verbs: - - create - - patch - - update - - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - labels: - kubernetes.io/cluster-service: "true" - name: system:azure-cloud-provider - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: system:azure-cloud-provider - subjects: - - kind: ServiceAccount - name: azure-cloud-provider - namespace: kube-system - ``` - - * If the clusterrole with same has already been provisioned (e.g. for accessing azurefile secrets), then the above yaml should be merged together, e.g. - - ```yaml - kind: List - apiVersion: v1 - items: - - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole - metadata: - labels: - kubernetes.io/cluster-service: "true" - name: system:azure-cloud-provider - rules: - - apiGroups: [""] - resources: ["events"] - verbs: - - create - - patch - - update - - apiGroups: [""] - resources: ["secrets"] - verbs: - - get - - create - - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - labels: - kubernetes.io/cluster-service: "true" - name: system:azure-cloud-provider - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: system:azure-cloud-provider - subjects: - - kind: ServiceAccount - name: azure-cloud-provider - namespace: kube-system - - kind: ServiceAccount - name: persistent-volume-binder - namespace: kube-system - ``` +* ACTION REQUIRED: The Node.Status.Volumes.Attached.DevicePath fields is deprecated for CSI volumes and will be unset in a future release ([#71095](https://github.com/kubernetes/kubernetes/pull/71095), [@msau42](https://github.com/msau42)) ### Other notable changes -* Update metrics-server to v0.3.1 ([#68746](https://github.com/kubernetes/kubernetes/pull/68746), [@DirectXMan12](https://github.com/DirectXMan12)) -* Upgrade kubeadm's version of docker support ([#68495](https://github.com/kubernetes/kubernetes/pull/68495), [@yuansisi](https://github.com/yuansisi)) -* fix a bug that overwhelming number of prometheus metrics are generated because $NAMESPACE is not replaced by string "{namespace}" ([#68530](https://github.com/kubernetes/kubernetes/pull/68530), [@wenjiaswe](https://github.com/wenjiaswe)) -* The feature gates `ReadOnlyAPIDataVolumes` and `ServiceProxyAllowExternalIPs`, deprecated since 1.10, have been removed and any references must be removed from command-line invocations. ([#67951](https://github.com/kubernetes/kubernetes/pull/67951), [@liggitt](https://github.com/liggitt)) -* Verify invalid secret/configmap/projected volumes before calling setup ([#68691](https://github.com/kubernetes/kubernetes/pull/68691), [@gnufied](https://github.com/gnufied)) -* Fix bug that caused `kubectl` commands to sometimes fail to refresh access token when running against GKE clusters. ([#66314](https://github.com/kubernetes/kubernetes/pull/66314), [@jlowdermilk](https://github.com/jlowdermilk)) -* Use KubeDNS by default in GCE setups, as CoreDNS has significantly higher memory usage in large clusters. ([#68629](https://github.com/kubernetes/kubernetes/pull/68629), [@shyamjvs](https://github.com/shyamjvs)) -* Fix PodAntiAffinity issues in case of multiple affinityTerms. ([#68173](https://github.com/kubernetes/kubernetes/pull/68173), [@Huang-Wei](https://github.com/Huang-Wei)) -* Make APIGroup field in TypedLocalObjectReference optional. ([#68419](https://github.com/kubernetes/kubernetes/pull/68419), [@xing-yang](https://github.com/xing-yang)) -* Fix potential panic when getting azure load balancer status ([#68609](https://github.com/kubernetes/kubernetes/pull/68609), [@feiskyer](https://github.com/feiskyer)) -* Fix kubelet panics when RuntimeClass is enabled. ([#68521](https://github.com/kubernetes/kubernetes/pull/68521), [@yujuhong](https://github.com/yujuhong)) -* - cAdvisor: Fix NVML initialization race condition ([#68431](https://github.com/kubernetes/kubernetes/pull/68431), [@dashpole](https://github.com/dashpole)) - * - cAdvisor: Fix brtfs filesystem discovery - * - cAdvisor: Fix race condition with AllDockerContainers - * - cAdvisor: Don't watch .mount cgroups - * - cAdvisor: Reduce lock contention during list containers -* Promote ScheduleDaemonSetPods by default scheduler to beta ([#67899](https://github.com/kubernetes/kubernetes/pull/67899), [@ravisantoshgudimetla](https://github.com/ravisantoshgudimetla)) +* Raw block volume support is promoted to beta, and enabled by default. This is accessible via the `volumeDevices` container field in pod specs, and the `volumeMode` field in persistent volume and persistent volume claims definitions. ([#71167](https://github.com/kubernetes/kubernetes/pull/71167), [@msau42](https://github.com/msau42)) +* Fix a scheduler panic due to internal cache inconsistency ([#71063](https://github.com/kubernetes/kubernetes/pull/71063), [@Huang-Wei](https://github.com/Huang-Wei)) +* Fix a potential bug that scheduler preempts unnecessary pods. ([#70898](https://github.com/kubernetes/kubernetes/pull/70898), [@Huang-Wei](https://github.com/Huang-Wei)) +* The API server encryption configuration file format has graduated to stable and moved to `apiVersion: apiserver.config.k8s.io/v1` and `kind: EncryptionConfiguration`. ([#67383](https://github.com/kubernetes/kubernetes/pull/67383), [@stlaz](https://github.com/stlaz)) +* kubelet now supports `log-file` option to write logs directly to a specific file ([#70917](https://github.com/kubernetes/kubernetes/pull/70917), [@dims](https://github.com/dims)) +* kubeadm now supports the `--image-repository` flag for customizing what registry to pull images from ([#71135](https://github.com/kubernetes/kubernetes/pull/71135), [@luxas](https://github.com/luxas)) +* timeouts set in ListOptions for clients will also be respected locally ([#70998](https://github.com/kubernetes/kubernetes/pull/70998), [@deads2k](https://github.com/deads2k)) +* IPVS proxier now set net/ipv4/vs/conn_reuse_mode to 0 by default, which will highly improve IPVS proxier performance. ([#71114](https://github.com/kubernetes/kubernetes/pull/71114), [@Lion-Wei](https://github.com/Lion-Wei)) +* StatefulSet is supported in `kubectl autoscale` command ([#71103](https://github.com/kubernetes/kubernetes/pull/71103), [@Pingan2017](https://github.com/Pingan2017)) +* Report kube-scheduler unhealthy if leader election is deadlocked. ([#71085](https://github.com/kubernetes/kubernetes/pull/71085), [@bsalamat](https://github.com/bsalamat)) +* apiserver: fixes handling and logging of panics in REST handlers ([#71076](https://github.com/kubernetes/kubernetes/pull/71076), [@liggitt](https://github.com/liggitt)) +* kubelets are no longer allowed to delete their own Node API object. Prior to 1.11, in rare circumstances related to cloudprovider node ID changes, kubelets would attempt to delete/recreate their Node object at startup. Kubelets older than 1.11 are not supported running against a v1.13+ API server. If an unsupported legacy kubelet encounters this situation, a cluster admin can remove the Node object: ([#71021](https://github.com/kubernetes/kubernetes/pull/71021), [@liggitt](https://github.com/liggitt)) + * `kubectl delete node/` + * or grant self-deletion permission explicitly: + * `kubectl create clusterrole self-deleting-nodes --verb=delete --resource=nodes` + * `kubectl create clusterrolebinding self-deleting-nodes --clusterrole=self-deleting-nodes --group=system:nodes` +* Kubernetes v1.13 moves support for Container Storage Interface to GA. As part of this move Kubernetes now supports CSI v1.0.0 and drops support for CSI 0.3 and older releases. Older CSI drivers must be updated to CSI 1.0 in order to work with Kubernetes 1.13+. ([#71020](https://github.com/kubernetes/kubernetes/pull/71020), [@saad-ali](https://github.com/saad-ali)) +* Remove deprecated kubectl command aliases 'run-container' ([#70728](https://github.com/kubernetes/kubernetes/pull/70728), [@Pingan2017](https://github.com/Pingan2017)) +* kubeadm: enable strict unmarshaling of YAML configuration files and show warnings for unknown and duplicate fields. ([#70901](https://github.com/kubernetes/kubernetes/pull/70901), [@neolit123](https://github.com/neolit123)) +* For kube-up and derived configurations, CoreDNS will honor master taints, for consistency with kube-dns behavior. ([#70868](https://github.com/kubernetes/kubernetes/pull/70868), [@justinsb](https://github.com/justinsb)) +* CoreDNS is now version 1.2.6 ([#70799](https://github.com/kubernetes/kubernetes/pull/70799), [@rajansandeep](https://github.com/rajansandeep)) +* kubeadm: Use `advertise-client-urls` instead of `listen-client-urls` as and `etcd-servers` options for apiserver. ([#69827](https://github.com/kubernetes/kubernetes/pull/69827), [@tomkukral](https://github.com/tomkukral)) +* Add option to create CSRs instead of certificates for kubeadm init phase certs and kubeadm alpha certs renew ([#70809](https://github.com/kubernetes/kubernetes/pull/70809), [@liztio](https://github.com/liztio)) +* Add a kubelet socket which serves an grpc service containing the devices used by containers on the node. ([#70508](https://github.com/kubernetes/kubernetes/pull/70508), [@dashpole](https://github.com/dashpole)) +* kube-apiserver: the `NodeRestriction` admission plugin now prevents kubelets from modifying `Node` labels prefixed with `node-restriction.kubernetes.io/`. The `node-restriction.kubernetes.io/` label prefix is reserved for cluster administrators to use for labeling `Node` objects to target workloads to nodes in a way that kubelets cannot modify or spoof. ([#68267](https://github.com/kubernetes/kubernetes/pull/68267), [@liggitt](https://github.com/liggitt)) + * kubelet: it is now deprecated to use the `--node-labels` flag to set `kubernetes.io/` and `k8s.io/`-prefixed labels other than the following labels: + * `kubernetes.io/hostname` + * `kubernetes.io/instance-type` + * `kubernetes.io/os` + * `kubernetes.io/arch` + * `beta.kubernetes.io/instance-type` + * `beta.kubernetes.io/os` + * `beta.kubernetes.io/arch` + * `failure-domain.kubernetes.io/zone` + * `failure-domain.kubernetes.io/region` + * `failure-domain.beta.kubernetes.io/zone` + * `failure-domain.beta.kubernetes.io/region` + * `[*.]kubelet.kubernetes.io/*` + * `[*.]node.kubernetes.io/*` + * Setting other `kubernetes.io/`- and `k8s.io/`-prefixed labels using the `--node-labels` flag will produce a warning in v1.13, and be disallowed in v1.15. Setting labels that are not prefixed with `kubernetes.io/` or `k8s.io/` is still permitted. +* Adds DynamicAuditing feature which allows for the configuration of audit webhooks through the use of an AuditSink API object. ([#67257](https://github.com/kubernetes/kubernetes/pull/67257), [@pbarker](https://github.com/pbarker)) +* The Kubelet plugin registration mechanism used by device plugins and CSI plugins is now GA ([#70559](https://github.com/kubernetes/kubernetes/pull/70559), [@vladimirvivien](https://github.com/vladimirvivien)) +* CSIPersistentVolume feature, i.e. PersistentVolumes with CSIPersistentVolumeSource, is GA. ([#69929](https://github.com/kubernetes/kubernetes/pull/69929), [@jsafrane](https://github.com/jsafrane)) + * CSIPersistentVolume feature gate is now deprecated and will be removed according to deprecation policy. +* kubectl: support multiple arguments for cordon/uncordon and drain ([#68655](https://github.com/kubernetes/kubernetes/pull/68655), [@goodluckbot](https://github.com/goodluckbot)) +* The kube-apiserver's healthz now takes in an optional query parameter which allows you to disable health checks from causing healthz failures. ([#70676](https://github.com/kubernetes/kubernetes/pull/70676), [@logicalhan](https://github.com/logicalhan)) +* client-go: fixes sending oversized data frames to spdystreams in remotecommand.NewSPDYExecutor ([#70999](https://github.com/kubernetes/kubernetes/pull/70999), [@liggitt](https://github.com/liggitt)) +* kube-controller-manager no longer removes ownerReferences from ResourceQuota objects ([#70035](https://github.com/kubernetes/kubernetes/pull/70035), [@liggitt](https://github.com/liggitt)) +* Introduces support for running a nodelocal dns cache. It is disabled by default, can be enabled by setting KUBE_ENABLE_NODELOCAL_DNS=true ([#70555](https://github.com/kubernetes/kubernetes/pull/70555), [@prameshj](https://github.com/prameshj)) + * An ip address is required for the cache instance to listen for requests on, default is a link local ip address of value 169.254.20.10 +* Fix dry-run output in kubectl apply --prune ([#69344](https://github.com/kubernetes/kubernetes/pull/69344), [@zegl](https://github.com/zegl)) +* kubectl run now generates apps/v1 deployments by default ([#71006](https://github.com/kubernetes/kubernetes/pull/71006), [@liggitt](https://github.com/liggitt)) +* `kubeadm reset` now outputs instructions about manual iptables rules cleanup. ([#70874](https://github.com/kubernetes/kubernetes/pull/70874), [@rdodev](https://github.com/rdodev)) +* Recognize newer docker versions without -ce/-ee suffix: 18.09.0 ([#71001](https://github.com/kubernetes/kubernetes/pull/71001), [@thomas-riccardi](https://github.com/thomas-riccardi)) +* "unfinished_work_microseconds" is added to the workqueue metrics; it can be used to detect stuck worker threads. (kube-controller-manager runs many workqueues.) ([#70884](https://github.com/kubernetes/kubernetes/pull/70884), [@lavalamp](https://github.com/lavalamp)) +* add readiness gates in extended output for pods ([#70775](https://github.com/kubernetes/kubernetes/pull/70775), [@freehan](https://github.com/freehan)) +* add `Ready` column and improve human-readable output of Deployments and StatefulSets ([#70466](https://github.com/kubernetes/kubernetes/pull/70466), [@Pingan2017](https://github.com/Pingan2017)) +* Kubeadm now respects the custom image registry configuration across joins and upgrades. Kubeadm passes the custom registry to the kubelet for a custom pause container. ([#70603](https://github.com/kubernetes/kubernetes/pull/70603), [@chuckha](https://github.com/chuckha)) +* kubeadm: deprecate the DynamicKubeletConfig feature gate. The functionality is still accessible by using the kubeadm alpha kubelet enable-dynamic command. ([#70849](https://github.com/kubernetes/kubernetes/pull/70849), [@yagonobre](https://github.com/yagonobre)) +* Add `kubelet_container_log_size_bytes` metric representing the log file size of a container. ([#70749](https://github.com/kubernetes/kubernetes/pull/70749), [@brancz](https://github.com/brancz)) +* kubeadm: remove the AuditPolicyConfiguration feature gate ([#70807](https://github.com/kubernetes/kubernetes/pull/70807), [@Klaven](https://github.com/Klaven)) +* Kubeadm: attributes for join --control-plane workflow are now grouped into a dedicated JoinControlPlane struct ([#70870](https://github.com/kubernetes/kubernetes/pull/70870), [@fabriziopandini](https://github.com/fabriziopandini)) +* Addon configuration is introduced in the kubeadm config API, while feature flag CoreDNS is now deprecated. ([#70024](https://github.com/kubernetes/kubernetes/pull/70024), [@fabriziopandini](https://github.com/fabriziopandini)) +* Fixes ability for admin/edit/view users to see controller revisions, needed for kubectl rollout commands ([#70699](https://github.com/kubernetes/kubernetes/pull/70699), [@liggitt](https://github.com/liggitt)) +* kubeadm pre-pulls Etcd image only if external Etcd is not used and ([#70743](https://github.com/kubernetes/kubernetes/pull/70743), [@bart0sh](https://github.com/bart0sh)) + * --etcd-upgrade=false is not specified +* Add support for CRD conversion webhook ([#67006](https://github.com/kubernetes/kubernetes/pull/67006), [@mbohlool](https://github.com/mbohlool)) +* Delete node lease if the corresponding node is deleted ([#70034](https://github.com/kubernetes/kubernetes/pull/70034), [@wangzhen127](https://github.com/wangzhen127)) +* In a future release the kubectl convert command will be deprecated. ([#70820](https://github.com/kubernetes/kubernetes/pull/70820), [@seans3](https://github.com/seans3)) +* kubeadm: UnifiedControlPlaneImage is replaced by UseHyperKubeImage boolean value. ([#70793](https://github.com/kubernetes/kubernetes/pull/70793), [@rosti](https://github.com/rosti)) +* kubeadm v1beta1 API: InitConfiguration.APIEndpoint has been renamed to .LocalAPIEndpoint ([#70761](https://github.com/kubernetes/kubernetes/pull/70761), [@luxas](https://github.com/luxas)) +* Breaking change: CSINodeInfo split into Spec and Status. New fields Available and VolumePluginMechanism added to CSINodeInfo csi-api object. CSIDriverInfo no longer deleted on Driver uninstallation, instead Available flag is set to false. ([#70515](https://github.com/kubernetes/kubernetes/pull/70515), [@davidz627](https://github.com/davidz627)) +* GCERegionalPersistentDisk feature is GA now! ([#70716](https://github.com/kubernetes/kubernetes/pull/70716), [@jingxu97](https://github.com/jingxu97)) +* Add secure port 10259 to the kube-scheduler (enabled by default) and deprecate old insecure port 10251. Without further flags self-signed certs are created on startup in memory. ([#69663](https://github.com/kubernetes/kubernetes/pull/69663), [@sttts](https://github.com/sttts)) +* `--feature-gates` argument has been removed from the `kubeadm join` command. Feature gates will be retrieved from the cluster configuration during the join process. ([#70755](https://github.com/kubernetes/kubernetes/pull/70755), [@ereslibre](https://github.com/ereslibre)) +* [kubeadm] Updates version of CoreDNS to 1.2.6 ([#70796](https://github.com/kubernetes/kubernetes/pull/70796), [@detiber](https://github.com/detiber)) +* kubelet: When node lease feature is enabled, kubelet reports node status to api server only if there is some change or it didn't report over last report interval. ([#69753](https://github.com/kubernetes/kubernetes/pull/69753), [@wangzhen127](https://github.com/wangzhen127)) +* Self hosted is no longer supported in the standard workflow. The feature flags have been removed and your self hosted cluster is no longer able to upgrade via kubeadm. ([#69878](https://github.com/kubernetes/kubernetes/pull/69878), [@Klaven](https://github.com/Klaven)) +* vSphereVolume implements Raw Block Volume Support ([#68761](https://github.com/kubernetes/kubernetes/pull/68761), [@fanzhangio](https://github.com/fanzhangio)) +* [GCE] Filter out spammy audit logs from cluster autoscaler. ([#70696](https://github.com/kubernetes/kubernetes/pull/70696), [@loburm](https://github.com/loburm)) +* CRD supports multi-version Schema, Subresources and AdditionalPrintColumns (NOTE that CRDs created prior to 1.13 populated the top-level additionalPrinterColumns field by default. To apply an update that changes to per-version additionalPrinterColumns, the top-level additionalPrinterColumns field must be explicitly set to null). ([#70211](https://github.com/kubernetes/kubernetes/pull/70211), [@roycaihw](https://github.com/roycaihw)) +* Fixes a bug in previous releases where a pod could be placed inside another pod's cgroup when specifying --cgroup-root ([#70678](https://github.com/kubernetes/kubernetes/pull/70678), [@dashpole](https://github.com/dashpole)) +* Upgrade golang.org/x/net image to release-branch.go1.10 ([#70663](https://github.com/kubernetes/kubernetes/pull/70663), [@wenjiaswe](https://github.com/wenjiaswe)) +* New addon in addon manager that automatically installs CSI CRDs if CSIDriverRegistry or CSINodeInfo feature gates are true. ([#70193](https://github.com/kubernetes/kubernetes/pull/70193), [@saad-ali](https://github.com/saad-ali)) +* delegated authorization can now allow unrestricted access for `system:masters` like the main kube-apiserver ([#70671](https://github.com/kubernetes/kubernetes/pull/70671), [@deads2k](https://github.com/deads2k)) +* Update to use go1.11.2 ([#70665](https://github.com/kubernetes/kubernetes/pull/70665), [@cblecker](https://github.com/cblecker)) +* Add dns capabilities for Windows CNI plugins: ([#67435](https://github.com/kubernetes/kubernetes/pull/67435), [@feiskyer](https://github.com/feiskyer)) + * "dns" { + * "servers": ["10.0.0.10"], + * "searches": ["default.svc.cluster.local","svc.cluster.local","cluster.local"], + * "options": [] + * } +* The VolumeScheduling feature is GA. The VolumeScheduling feature gate is deprecated and will be removed in a future release. ([#70673](https://github.com/kubernetes/kubernetes/pull/70673), [@msau42](https://github.com/msau42)) +* Go clients created from a kubeconfig that specifies a TokenFile now periodically reload the token from the specified file. ([#70606](https://github.com/kubernetes/kubernetes/pull/70606), [@mikedanese](https://github.com/mikedanese)) +* kubeadm: validate kubeconfig files in case of external CA mode. ([#70537](https://github.com/kubernetes/kubernetes/pull/70537), [@yagonobre](https://github.com/yagonobre)) +* kube-apiserver: `--audit-webhook-version` and `--audit-log-version` now default to `audit.k8s.io/v1` if unspecified ([#70476](https://github.com/kubernetes/kubernetes/pull/70476), [@charrywanganthony](https://github.com/charrywanganthony)) +* kubeadm: timeoutForControlPlane is introduced as part of the API Server config, that controls the timeout for the wait for control plane to be up. Default value is 4 minutes. ([#70480](https://github.com/kubernetes/kubernetes/pull/70480), [@rosti](https://github.com/rosti)) +* kubeadm: The writable config file option for extra volumes is renamed to readOnly with a reversed meaning. With readOnly defaulted to false (as in pod specs). ([#70495](https://github.com/kubernetes/kubernetes/pull/70495), [@rosti](https://github.com/rosti)) +* remove retry operation on attach/detach azure disk ([#70568](https://github.com/kubernetes/kubernetes/pull/70568), [@andyzhangx](https://github.com/andyzhangx)) +* Fix CSI volume limits not showing up in node's capacity and allocatable ([#70540](https://github.com/kubernetes/kubernetes/pull/70540), [@gnufied](https://github.com/gnufied)) +* Flex volume plugins now support expandvolume (to increase underlying volume capacity) and expanfs (resize filesystem) commands that Flex plugin authors can implement to support expanding in use Flex PersistentVolumes ([#67851](https://github.com/kubernetes/kubernetes/pull/67851), [@aniket-s-kulkarni](https://github.com/aniket-s-kulkarni)) +* kubeadm: Control plane component configs are separated into ClusterConfiguration sub-structs. ([#70371](https://github.com/kubernetes/kubernetes/pull/70371), [@rosti](https://github.com/rosti)) +* The `MountPropagation` feature is unconditionally enabled in v1.13, and can no longer be disabled. ([#68230](https://github.com/kubernetes/kubernetes/pull/68230), [@bertinatto](https://github.com/bertinatto)) +* add azure UltraSSD, StandardSSD disk type support ([#70477](https://github.com/kubernetes/kubernetes/pull/70477), [@andyzhangx](https://github.com/andyzhangx)) +* The OwnerReferencesPermissionEnforcement admission plugin now checks authorization for the correct scope (namespaced or cluster-scoped) of the owner resource type. Previously, it always checked permissions at the same scope as the child resource. ([#70389](https://github.com/kubernetes/kubernetes/pull/70389), [@caesarxuchao](https://github.com/caesarxuchao)) +* Ensure orphan public IPs on Azure deleted when service recreated with the same name. ([#70463](https://github.com/kubernetes/kubernetes/pull/70463), [@feiskyer](https://github.com/feiskyer)) +* `kubectl apply` can now change a deployment strategy from rollout to recreate without explicitly clearing the rollout-related fields ([#70436](https://github.com/kubernetes/kubernetes/pull/70436), [@liggitt](https://github.com/liggitt)) +* Fix cloud-controller-manager crash when using OpenStack provider and PersistentVolume initializing controller ([#70459](https://github.com/kubernetes/kubernetes/pull/70459), [@mvladev](https://github.com/mvladev)) -# v1.12.0-beta.2 +# v1.13.0-alpha.3 -[Documentation](https://docs.k8s.io) & [Examples](https://releases.k8s.io/release-1.12/examples) +[Documentation](https://docs.k8s.io) -## Downloads for v1.12.0-beta.2 +## Downloads for v1.13.0-alpha.3 -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes.tar.gz) | `7163d18b9c1bd98ce804b17469ed67b399deb7b574dd12a86609fc647c5c773b` -[kubernetes-src.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-src.tar.gz) | `6225b71b2dec0f29afb713e64d2b6b82bd0e122274c31310c0de19ef023cb1d0` +[kubernetes.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes.tar.gz) | `1d50cfd34306ace7354516125c45f8c546bba3ca5081af2b21969b535967d302821c06e7d4d590ba05f3e1e89ba56cc3d890b2e5bf2e07456532ac28e7c6c4a8` +[kubernetes-src.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-src.tar.gz) | `bf097b99d7b9af15bc1d592ee3782da1e811d8eb68dc9ae9d287589ce9174d3743beaf51422283c42ad03775e43839954bdeb44b1aa5707f73eebabed0cc199e` ### Client Binaries -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-client-darwin-386.tar.gz) | `f2ec9799e47c28fce336bc90a6e9b4e47def7081fd73b8e2164940f0a6c824c7` -[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-client-darwin-amd64.tar.gz) | `0e8cfcbe5ec862423ced97da1d9740d4cc4904a0d5cd11a60616aee596bc7622` -[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-client-linux-386.tar.gz) | `1cbd6e8dd892cfc2555d37e733b66aaf85df9950466c7295875d312ac254ddfc` -[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-client-linux-amd64.tar.gz) | `47337b58a26a4953e5c061d28e3ec89b3d4354bce40f9b51fbe269598caeff03` -[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-client-linux-arm.tar.gz) | `eaaed82f428fb7ddbb10b4e39a2f287817c33ae24ff16008159f437acc653d4a` -[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-client-linux-arm64.tar.gz) | `3249d1c7d5d5500793546eb144fe537d1984a01c7a79c1382eb2e26a78e532cd` -[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-client-linux-ppc64le.tar.gz) | `67afd34f2199deff901b0872a177dc448ba700dc4ced9ede6f3187a0eed2c6fb` -[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-client-linux-s390x.tar.gz) | `e8faa6e45c6e2aeb67ac65737e09be87c190e3c89782ec87a9a205d4f1af9246` -[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-client-windows-386.tar.gz) | `2395051c8cbd0a995b5f3689c0f8c0447bcc1c46440d8cdeffd7c7fccf8e8ae1` -[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-client-windows-amd64.tar.gz) | `c6a38ee6eda20656b391ecfcc1f24505eb8a3a5a3200d4bddede318291773619` +[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-client-darwin-386.tar.gz) | `77778ae2887eda52ee716fb0e843c17b2705b1284a67cdf53f91292eb7f1055ef942be1ae0eac25ad9f4c9062c802fd57943e57578113f875f0d1f8a227bbb25` +[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-client-darwin-amd64.tar.gz) | `b3399767df12b71ee4b7b30126bd8001a0c1396161eb7535d797fd5847c55bccc18fb475f1639c3bf8e5f5c484a61025108c01e35f71417a405e75580c3990f6` +[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-client-linux-386.tar.gz) | `5ef0d318ff8da28c332ae25164e5a441272d2ee8ef2ac26438a47fe3e7e645ed0b132325243f3f33c93a8681431a117f6207893f0e5270738aacfee16d8bbd32` +[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-client-linux-amd64.tar.gz) | `1f429eae5b0b1e39b1d4d30e3220a82d0ae6672a6f4b34a05246c3efc131a236f79e5876d7a98c56d98fbf243b96cfd8db5d1ab73905df9e000d7af2b710fcf3` +[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-client-linux-arm.tar.gz) | `5583aecdc9b4a54a4aa904fc1de66400f50628969e31b5a63ab1d3b6628e3c547a9cef9c2bc9e53a0eeb0155fa0f745f7ee47e92166ce66d0c79199ff20c41de` +[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-client-linux-arm64.tar.gz) | `2453b9100c06b11e8c424d59cfd1c5e111c22b596191a9cfb0b330d198abecd982e19eb2ac38d91c3d1ef226c9533c4f9a210b88295029985739e86363905de5` +[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-client-linux-ppc64le.tar.gz) | `4991ec4c19a82d50caed78bc8db51e7cdcd1f2896dfcaa45d84f347a72fe7ee0d0280c897af54afec93f548d2ce247cf70d2b439024752ff701b80b67c385913` +[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-client-linux-s390x.tar.gz) | `c55f2802afb2e5d261bb26b6c396df8ebe6b95913ddab1e124cf177f59a00524a26cc9dd74cbf06cc7a6a2207ccba2d3a8585f20a37485214d9b1bebb27fdde7` +[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-client-windows-386.tar.gz) | `df78465267e35ef078c3c0fd33f8898a9df26fbf411df3ed3283fbdc2e79380693abe2a2815d5a4d53742135db4b5a0a480ce301817ede5c0674b12ac9c9bac9` +[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-client-windows-amd64.tar.gz) | `5b93fdaaa931ef8e24196e53c484f91ef9e50b7d11e1053ccb61b2d6bdc8164767dd1d885cdc4949fdb7ed05cfbc7c9f7cc3468ad7affc22f849836b311d855f` ### Server Binaries -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-server-linux-amd64.tar.gz) | `795c713a91118218f5952e1bd4cf0933f36476aa3d9d60a9ee43c9bae8400fd3` -[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-server-linux-arm.tar.gz) | `1798d48a37b8f06878e0ecb8d9b67d0fb5c8ee721608412add57725eb5ce5f1e` -[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-server-linux-arm64.tar.gz) | `da2459b5e811daaa2fc04a072773e81dc220400f3aeb6e29bb9594c306c7b266` -[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-server-linux-ppc64le.tar.gz) | `7fd1c2ba0c2c9da5db54f8d0aed28261f03e9953ce01fa367e4ce3d84bf01b4f` -[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-server-linux-s390x.tar.gz) | `c9fafb009d7e5da74f588aaa935244c452de52b9488863b90e8b477b1bb16e52` +[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-server-linux-amd64.tar.gz) | `922a93ce677e686e594c11db75e1969c995b23062bba511bff4a43d3a530e21e1d15cbe9b38d06af407de5a6c141f3b10cf24f50b848ed6b36e701249cece2c3` +[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-server-linux-arm.tar.gz) | `5dd550b58dedf25df020e66f1526e80c50b46d2df3ddd241bd02b6ebf10308599e7739917cbd9a3d2d3e787078d1271c4e3333f7f343a8a754ac5c08dedba0c9` +[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-server-linux-arm64.tar.gz) | `3e1037e71d85a74cd5d40dd836bd442b2dcc457f8ccc8247e4537f3deca6f99d6805857c44b4fd69a7655308a3b0b4be43faaf74017cc3e26e8ba81e84968f33` +[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-server-linux-ppc64le.tar.gz) | `a89c46b558613ad09efe44a81574ad18157a787d1e9c5d09c98d3911b499573cd9b9845b7a7526d4de5b93887267bedd96d043f79799552639b2168d3fea49ab` +[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-server-linux-s390x.tar.gz) | `47a68668e38ac1b8cb801f4bff3b15060cd88801f446ebfbf06125dbc9aef52be79faa94acee65af46b93512af6f15c1f9f598324a368b82996f5a0f9aca0d14` ### Node Binaries -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-node-linux-amd64.tar.gz) | `ab901137b499829b20b868492d04c1f69d738620b96eb349c642d6d773c44448` -[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-node-linux-arm.tar.gz) | `116dd82721f200f3f37df0e47aebb611fdd7856f94d4c2ebb1d51db21b793a9c` -[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-node-linux-arm64.tar.gz) | `56d8316eb95f7f54c154625063617b86ffb8e2cc80b8225cce4f5c91d2d3a64f` -[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-node-linux-ppc64le.tar.gz) | `66535b16ad588ba3bfcb40728a0497c6821360ab7be9c3ced2072bfa107e5c46` -[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-node-linux-s390x.tar.gz) | `688e09becc9327e50c68b33161eac63a8ba018c02fb298cbd0de82d6ed5dba90` -[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.12.0-beta.2/kubernetes-node-windows-amd64.tar.gz) | `b72582f67d19c06f605ca9b02c08b7227796c15c639e3c09b06a8b667c4569fe` +[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-node-linux-amd64.tar.gz) | `74d5d46ac6ba336fa8aaf55d0a15860f6ebde2ff58d377ca93063593da12fbd4c758fe92c6b3285df0e4cc23c354ce5802ae3f7c38ad4e63f9d1eeaef41f2f2b` +[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-node-linux-arm.tar.gz) | `90372cb5270ffe6179d5d7efd3fff6aa029f73853805038fef1a6f92683399d692aee7790eacc3a59681f66e2bd21b76c3abd87bef6404382782ec82fb726d59` +[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-node-linux-arm64.tar.gz) | `09693303a1a8489d9599d32f7fbf549d18f31eb53671fa2ed342fe5089f2fba9322d8c52088b141da2414bb2b7f8352634cf84f0c72f3a58afd057629c3e8e5b` +[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-node-linux-ppc64le.tar.gz) | `cbdb3b9ffd9be524ec0b38d72b0545b6dd1b3b789747f41a661fd7cbeffe942eb592c7e928a61e26152e7402b4894f5a74aad09cf2243d726b3b311d41cb0674` +[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-node-linux-s390x.tar.gz) | `fc296b386bc03bf10773559118cd4a3d5be3d4c296f09748507fac812a1c791b435881b2ae425846632404f7f53f3f3bf54fc582fb8aa4895310186371dbacfd` +[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.3/kubernetes-node-windows-amd64.tar.gz) | `ae79c62fcb0654a62606d65cf131188d93e4a10787a862e7b0363269942df19543b55fd119ec6ff73578b24db7774235611956103fe5e153eddf450750c4bbee` -## Changelog since v1.12.0-beta.1 - -### Action Required - -* Action required: The --storage-versions flag of kube-apiserver is deprecated. Please omit this flag to ensure the default storage versions are used. Otherwise the cluster is not safe to upgrade to a version newer than 1.12. This flag will be removed in 1.13. ([#68080](https://github.com/kubernetes/kubernetes/pull/68080), [@caesarxuchao](https://github.com/caesarxuchao)) +## Changelog since v1.13.0-alpha.2 ### Other notable changes -* kubeadm: add mandatory "--config" flag to "kubeadm alpha phase preflight" ([#68446](https://github.com/kubernetes/kubernetes/pull/68446), [@neolit123](https://github.com/neolit123)) -* Apply user configurations for local etcd ([#68334](https://github.com/kubernetes/kubernetes/pull/68334), [@SataQiu](https://github.com/SataQiu)) -* kubeadm: added phase command "alpha phase kubelet config annotate-cri" ([#68449](https://github.com/kubernetes/kubernetes/pull/68449), [@fabriziopandini](https://github.com/fabriziopandini)) -* If `TaintNodesByCondition` is enabled, add `node.kubernetes.io/unschedulable` and ([#64954](https://github.com/kubernetes/kubernetes/pull/64954), [@k82cn](https://github.com/k82cn)) - * `node.kubernetes.io/network-unavailable` automatically to DaemonSet pods. -* Deprecate cloudstack and ovirt controllers ([#68199](https://github.com/kubernetes/kubernetes/pull/68199), [@dims](https://github.com/dims)) -* add missing LastTransitionTime of ContainerReady condition ([#64867](https://github.com/kubernetes/kubernetes/pull/64867), [@dixudx](https://github.com/dixudx)) -* kube-controller-manager: use informer cache instead of active pod gets in HPA controller ([#68241](https://github.com/kubernetes/kubernetes/pull/68241), [@krzysztof-jastrzebski](https://github.com/krzysztof-jastrzebski)) -* Support NodeShutdown taint for azure ([#68033](https://github.com/kubernetes/kubernetes/pull/68033), [@yastij](https://github.com/yastij)) -* Registers volume topology information reported by a node-level Container Storage Interface (CSI) driver. This enables Kubernetes support of CSI topology mechanisms. ([#67684](https://github.com/kubernetes/kubernetes/pull/67684), [@verult](https://github.com/verult)) -* Update default etcd server to 3.2.24 for kubernetes 1.12 ([#68318](https://github.com/kubernetes/kubernetes/pull/68318), [@timothysc](https://github.com/timothysc)) -* External CAs can now be used for kubeadm with only a certificate, as long as all required certificates already exist. ([#68296](https://github.com/kubernetes/kubernetes/pull/68296), [@liztio](https://github.com/liztio)) -* Bump addon-manager to v8.7 ([#68299](https://github.com/kubernetes/kubernetes/pull/68299), [@MrHohn](https://github.com/MrHohn)) - * - Support extra `--prune-whitelist` resources in kube-addon-manager. - * - Update kubectl to v1.10.7. -* Let service controller retry creating load balancer when persistUpdate failed due to conflict. ([#68087](https://github.com/kubernetes/kubernetes/pull/68087), [@grayluck](https://github.com/grayluck)) -* Kubelet now only sync iptables on Linux. ([#67690](https://github.com/kubernetes/kubernetes/pull/67690), [@feiskyer](https://github.com/feiskyer)) -* CSI NodePublish call can optionally contain information about the pod that requested the CSI volume. ([#67945](https://github.com/kubernetes/kubernetes/pull/67945), [@jsafrane](https://github.com/jsafrane)) -* [e2e] verifying LimitRange update is effective before creating new pod ([#68171](https://github.com/kubernetes/kubernetes/pull/68171), [@dixudx](https://github.com/dixudx)) -* cluster/gce: generate consistent key sizes in config-default.sh using /dev/urandom instead of /dev/random ([#67139](https://github.com/kubernetes/kubernetes/pull/67139), [@yogi-sagar](https://github.com/yogi-sagar)) -* Add support for volume attach limits for CSI volumes ([#67731](https://github.com/kubernetes/kubernetes/pull/67731), [@gnufied](https://github.com/gnufied)) -* CSI volume plugin does not need external attacher for non-attachable CSI volumes. ([#67955](https://github.com/kubernetes/kubernetes/pull/67955), [@jsafrane](https://github.com/jsafrane)) -* KubeletPluginsWatcher feature graduates to beta. ([#68200](https://github.com/kubernetes/kubernetes/pull/68200), [@RenaudWasTaken](https://github.com/RenaudWasTaken)) -* Update etcd client to 3.2.24 for latest release ([#68147](https://github.com/kubernetes/kubernetes/pull/68147), [@timothysc](https://github.com/timothysc)) -* [fluentd-gcp-scaler addon] Bump fluentd-gcp-scaler to 0.4 to pick up security fixes. ([#67691](https://github.com/kubernetes/kubernetes/pull/67691), [@loburm](https://github.com/loburm)) - * [prometheus-to-sd addon] Bump prometheus-to-sd to 0.3.1 to pick up security fixes, bug fixes and new features. - * [event-exporter addon] Bump event-exporter to 0.2.3 to pick up security fixes. -* Fixes issue where pod scheduling may fail when using local PVs and pod affinity and anti-affinity without the default StatefulSet OrderedReady pod management policy ([#67556](https://github.com/kubernetes/kubernetes/pull/67556), [@msau42](https://github.com/msau42)) -* Kubelet only applies default hard evictions of nodefs.inodesFree on Linux ([#67709](https://github.com/kubernetes/kubernetes/pull/67709), [@feiskyer](https://github.com/feiskyer)) -* Add kubelet stats for windows system container "pods" ([#66427](https://github.com/kubernetes/kubernetes/pull/66427), [@feiskyer](https://github.com/feiskyer)) -* Add a TTL machenism to clean up Jobs after they finish. ([#66840](https://github.com/kubernetes/kubernetes/pull/66840), [@janetkuo](https://github.com/janetkuo)) +* kubelet --system-reserved and --kube-reserved are supported now on Windows nodes ([#69960](https://github.com/kubernetes/kubernetes/pull/69960), [@feiskyer](https://github.com/feiskyer)) +* CSI drivers now have access to mountOptions defined on the storage class when attaching volumes. ([#67898](https://github.com/kubernetes/kubernetes/pull/67898), [@bswartz](https://github.com/bswartz)) +* The `kubectl plugin list` command will now display discovered plugin paths in the same order as they are found in a user's PATH variable. ([#70443](https://github.com/kubernetes/kubernetes/pull/70443), [@juanvallejo](https://github.com/juanvallejo)) +* Handle Windows named pipes in host mounts. ([#69484](https://github.com/kubernetes/kubernetes/pull/69484), [@ddebroy](https://github.com/ddebroy)) +* kubeadm: Multiple API server endpoints support upon join is removed as it is now redundant. ([#69812](https://github.com/kubernetes/kubernetes/pull/69812), [@rosti](https://github.com/rosti)) +* OpenAPI spec marks delete request's body parameter as optional ([#70032](https://github.com/kubernetes/kubernetes/pull/70032), [@iamneha](https://github.com/iamneha)) +* kube-controller-manager and cloud-controller-manager now hold generated serving certificates in-memory unless a writeable location is specified with --cert-dir ([#69884](https://github.com/kubernetes/kubernetes/pull/69884), [@liggitt](https://github.com/liggitt)) +* Scheduler only activates unschedulable pods if node's scheduling related properties change. ([#70366](https://github.com/kubernetes/kubernetes/pull/70366), [@mlmhl](https://github.com/mlmhl)) +* --api-audiences now defaults to the --service-account-issuer if the issuer is provided but the API audience is not. ([#70308](https://github.com/kubernetes/kubernetes/pull/70308), [@mikedanese](https://github.com/mikedanese)) +* Refactor scheduler_test.go to use a fake k8s client. ([#70290](https://github.com/kubernetes/kubernetes/pull/70290), [@tossmilestone](https://github.com/tossmilestone)) +* `kubectl rollout undo` now returns errors when attempting to rollback a deployment to a non-existent revision ([#70039](https://github.com/kubernetes/kubernetes/pull/70039), [@liggitt](https://github.com/liggitt)) + * `kubectl rollout undo` no longer uses the deprecated extensions/v1beta1 rollback API, which means that Events are no longer emitted when rolling back a deployment +* - The builtin system:csi-external-provisioner and system:csi-external-attacher cluster roles ([#69868](https://github.com/kubernetes/kubernetes/pull/69868), [@pohly](https://github.com/pohly)) + * are deprecated and will not be updated for deployments of CSI sidecar container versions >= 0.4. + * Deployments with the current CSI sidecar containers have to provide their own RBAC + * definitions. The reason is that the rules depend on how the sidecar containers are used, + * which is defined by the deployment. +* Use debian-base instead of busybox as base image for server images ([#70245](https://github.com/kubernetes/kubernetes/pull/70245), [@ixdy](https://github.com/ixdy)) +* add support for projected volume in describe function ([#70158](https://github.com/kubernetes/kubernetes/pull/70158), [@WanLinghao](https://github.com/WanLinghao)) +* Speedup process lookup in /proc ([#66367](https://github.com/kubernetes/kubernetes/pull/66367), [@cpuguy83](https://github.com/cpuguy83)) +* Kubeadm reset now clean up custom etcd data path ([#70003](https://github.com/kubernetes/kubernetes/pull/70003), [@yagonobre](https://github.com/yagonobre)) +* We changed when the `metadata.generation` of a custom resource (CR) increments. ([#69059](https://github.com/kubernetes/kubernetes/pull/69059), [@caesarxuchao](https://github.com/caesarxuchao)) + * If the CR participates the spec/status convention, the metadata.generation of the CR increments when there is any change, except for the changes to the metadata or the changes to the status. + * If the CR does not participate the spec/status convention, the metadata.generation of the CR increments when there is any change to the CR, except for changes to the metadata. + * A CR is considered to participate the spec/status convention if and only if the "CustomResourceSubresources" feature gate is turned on and the CRD has `.spec.subresources.status={}`. +* Improve Azure instance metadata handling by adding caches. ([#70353](https://github.com/kubernetes/kubernetes/pull/70353), [@feiskyer](https://github.com/feiskyer)) +* adding cn-northwest-1 for AWS China Ningxia region ([#70155](https://github.com/kubernetes/kubernetes/pull/70155), [@pahud](https://github.com/pahud)) +* "kubectl get" no longer exits before printing all of its results if an error is found ([#70311](https://github.com/kubernetes/kubernetes/pull/70311), [@juanvallejo](https://github.com/juanvallejo)) +* kubeadm now automatically creates a new stacked etcd member when joining a new control plane node (does not applies to external etcd) ([#69486](https://github.com/kubernetes/kubernetes/pull/69486), [@fabriziopandini](https://github.com/fabriziopandini)) +* Critical pod annotation is deprecated. Pod priority should be used instead to mark pods as critical. ([#70298](https://github.com/kubernetes/kubernetes/pull/70298), [@bsalamat](https://github.com/bsalamat)) +* Display the usage of ephemeral-storage when using `kubectl describe node` ([#70268](https://github.com/kubernetes/kubernetes/pull/70268), [@Pingan2017](https://github.com/Pingan2017)) +* Added functionality to enable br_netfilter and ip_forward for debian packages to improve kubeadm support for CRI runtime besides Docker. ([#70152](https://github.com/kubernetes/kubernetes/pull/70152), [@ashwanikhemani](https://github.com/ashwanikhemani)) +* Add regions ap-northeast-3 and eu-west-3 to the list of well known AWS regions. ([#70252](https://github.com/kubernetes/kubernetes/pull/70252), [@nckturner](https://github.com/nckturner)) +* Remove kube-controller-manager flag '--insecure-experimental-approve-all-kubelet-csrs-for-group'(deprecated in v1.7) ([#69209](https://github.com/kubernetes/kubernetes/pull/69209), [@Pingan2017](https://github.com/Pingan2017)) +* GCE/GKE load balancer health check default interval changes from 2 seconds to 8 seconds, unhealthyThreshold to 3. ([#70099](https://github.com/kubernetes/kubernetes/pull/70099), [@grayluck](https://github.com/grayluck)) + * Health check parameters are configurable to be bigger than default values. +* The kubectl wait command must handle when a watch returns an error vs closing by printing out the error and retrying the watch. ([#69389](https://github.com/kubernetes/kubernetes/pull/69389), [@smarterclayton](https://github.com/smarterclayton)) +* Updates to use debian-iptables v11.0, debian-hyperkube-base 0.12.0, and kube-addon-manager:v8.9. ([#70209](https://github.com/kubernetes/kubernetes/pull/70209), [@ixdy](https://github.com/ixdy)) +* Fixed patch/update operations on multi-version custom resources ([#70087](https://github.com/kubernetes/kubernetes/pull/70087), [@liggitt](https://github.com/liggitt)) +* When `--rotate-server-certificates` is enabled, kubelet will no longer request a new certificate on startup if the current certificate on disk is satisfactory. ([#69991](https://github.com/kubernetes/kubernetes/pull/69991), [@agunnerson-ibm](https://github.com/agunnerson-ibm)) +* - Support for passing unknown provider names to the E2E test binaries is going to be deprecated. Use `--provider=skeleton` (no ssh access) or `--provider=local` (local cluster with ssh) instead. ([#70141](https://github.com/kubernetes/kubernetes/pull/70141), [@pohly](https://github.com/pohly)) +* Add scheduler benchmark tests for PodAffinity and NodeAffinity. ([#69898](https://github.com/kubernetes/kubernetes/pull/69898), [@Huang-Wei](https://github.com/Huang-Wei)) +* fix azure disk attachment error on Linux ([#70002](https://github.com/kubernetes/kubernetes/pull/70002), [@andyzhangx](https://github.com/andyzhangx)) -# v1.12.0-beta.1 +# v1.13.0-alpha.2 -[Documentation](https://docs.k8s.io) & [Examples](https://releases.k8s.io/release-1.12/examples) +[Documentation](https://docs.k8s.io) -## Downloads for v1.12.0-beta.1 +## Downloads for v1.13.0-alpha.2 -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes.tar.gz) | `caa332b14a6ea9d24710e3b015a91b62c04cab14bed14c49077e08bd82b8f4c1` -[kubernetes-src.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-src.tar.gz) | `821bdea3a52a348306fa8226bcfffa67b375cf1dd80e4be343ce0b38dd20a9a0` +[kubernetes.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes.tar.gz) | `cbe7ef29c7e7bbed82e173289f5f84d7a85ee4965cc5b7ccd16cf8236a3b8171bb8f52011d00ea4dd1119cfeee32a3326260e968038431f2834f194e84f6f070` +[kubernetes-src.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-src.tar.gz) | `8b0b8e1b635cd849c2974d755fe174f0ce8fe8c690721d8ac6312683bbd2ca2c6f7eada38e4e470d3a0172138b10a994fa3bebc01abd76d835431660247e8a71` ### Client Binaries -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-client-darwin-386.tar.gz) | `58323c0a81afe53dd0dda1c6eb513caa4c82514fb6c7f0a327242e573ce80490` -[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-client-darwin-amd64.tar.gz) | `28e9344ede16890ea7848c261e461ded89c3bb2dd5b08446da04b071b48f0b02` -[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-client-linux-386.tar.gz) | `a9eece5e0994d2ad5e07152d88787a8b5e9efcdf78983a5bafe3699e5274a9da` -[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-client-linux-amd64.tar.gz) | `9a67750cc4243335f0c2eb89db1c4b54b0a8af08c59e2041636d0a3e946546bf` -[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-client-linux-arm.tar.gz) | `bbd2644f843917a3de517a53c90b327502b577fe533a9ad3da4fe6bc437c4a02` -[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-client-linux-arm64.tar.gz) | `630946f49ef18dd43c004d99dccd9ae76390281f54740d7335c042f6f006324b` -[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-client-linux-ppc64le.tar.gz) | `1d4e5cd83faf4cae8e16667576492fcd48a72f69e8fd89d599a8b555a41e90d6` -[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-client-linux-s390x.tar.gz) | `9cefdcf21a62075b5238fda8ef2db08f81b0541ebce0e67353af1dded9e53483` -[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-client-windows-386.tar.gz) | `8b0085606ff38bded362bbe4826b5c8ee5199a33d5cbbc1b9b58f1336648ad5b` -[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-client-windows-amd64.tar.gz) | `f44a3ec55dc7d926e681c33b5f7830c6d1cb165e24e349e426c1089b2d05a1df` +[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-client-darwin-386.tar.gz) | `fca661a5001e7f368374d0805f20910be24baa485bf4ae5d993185b974f70ff7241497e7a130658dca69abf335fd581c0f5be22de4f9937f32e9fb20fbd0f1e9` +[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-client-darwin-amd64.tar.gz) | `d31dfea475981c7f7b758c7f201aa5b866db48d87942c79d0a12d464b7cdf501dba2255282c72b53928ccac215282fc8c0a9069a7443e9683bbf39a59814d8c3` +[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-client-linux-386.tar.gz) | `fecf8362c572fff48952fd2748ddcb9d375462cb484670cda4fda1387eb692713be0a323e93746bb1845a4ef3b15df288430c8b4a5e4f2892c388bb84a66ec9a` +[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-client-linux-amd64.tar.gz) | `136cb82ac94bcd791d56e997a948a7e1bee4af03bcc69ce9c835895cdda75524f5916756c778ff8aa693624289824eaee4aff1e62a6d95d18f9b69a7473a3fe9` +[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-client-linux-arm.tar.gz) | `e561c37895edef44614ecd59f497d393275ee62455b6269b169a891873d661d46cbf68e6148447f142ebdddb45bd5c01f6927d8463e2e1c4072358fec205e53e` +[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-client-linux-arm64.tar.gz) | `c0d5eb49763e8bf50b5e8e3785c7889fecbd8bf7c0b3c18250fa894a1c5e58a14f796f7526279dbf41d5d47e69114d39e37a62403ff4abb1cc0cd20bd48c1c5a` +[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-client-linux-ppc64le.tar.gz) | `a5a8c150af163e7c726662eeddfc3de8e43f123daaa100b8e82c9bc786313a5ce8135cbaadd41ebbb6c2307aeebdadf43fc84037a5ac9d3c5d6e9052b7fc615b` +[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-client-linux-s390x.tar.gz) | `fd162e0244e107f1892d79029f3452cdba84d8616ad1b15eebe197afb3b536328cd8cba9c73c0ce1cd8ac6faed9e0a4613d487bf07b45341fadf0d940062bafa` +[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-client-windows-386.tar.gz) | `e01fedec8f700e037bc43cb13bc916b85601cd1c9361a0f63fd27092640f89d2faaff4a6157638af8af52538c5f7fa75d9c7674477a45e0215fd505095a62f21` +[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-client-windows-amd64.tar.gz) | `d2601efcfa6a4ba8a017e9cac571fb454b21b7700a7b3f8e2fbabdd5301545ef2459da93eca3684c1aa73ea6a6d87bae0e27fffc8b05ed93729004ebbb9e3879` ### Server Binaries -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-server-linux-amd64.tar.gz) | `1bf7364aa168fc251768bc850d66fef1d93f324f0ec85f6dce74080627599b70` -[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-server-linux-arm.tar.gz) | `dadc94fc0564cfa98add5287763bbe9c33bf8ba3eebad95fb2258c33fe8c5df3` -[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-server-linux-arm64.tar.gz) | `2e6c8a7810705594f191b33476bf4c8fca8cebb364f0855dfea577b01fca7b7e` -[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-server-linux-ppc64le.tar.gz) | `ced4a0a4e03639378eff0d3b8bfb832f5fb96be8df3e0befbdbd71373a323130` -[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-server-linux-s390x.tar.gz) | `7e1a3fac2115c15b5baa0db04c7f319fbaaca92aa4c4588ecf62fb19812465a8` +[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-server-linux-amd64.tar.gz) | `4dda298d44bc309f250c067e9282eea37903838a140cf5abf6f861dca624d5a055ba1c43129454ae9abc8350f1fffb224677e34ef4268f7eb43c96fe48b28d78` +[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-server-linux-arm.tar.gz) | `e9c3bdf60272399bc6f85a15bbc55cd69db389c223b275661ddcab4ae8c3afcd2171ec3c35a53df2420376f8c6294af791776e16dc07a27438aaa3220d6c94a3` +[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-server-linux-arm64.tar.gz) | `d0a1701a34365f939799b6ea676129acdcfa1582bcf50e82a9751d9aafc73ebffd0fc0365fe352755caa9a2b791c4195d7eb04fbb1ce660f107cb5257d57a967` +[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-server-linux-ppc64le.tar.gz) | `1a23473960aaaa639e796020741b63c11dad8a93903926e80c871814b8209166ef06f5205172cd9b2438f8c8e9c9fb50b96d983f56737ab252530112f816b2d6` +[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-server-linux-s390x.tar.gz) | `293d0eb93e2ed641d0c1e26d58423670c04c307dddb034a9fc252043abe5694f63e772d4dfe2bf505fc0b68c41f9d87800d867b87c7697dd6249b11c0efbc580` ### Node Binaries -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-node-linux-amd64.tar.gz) | `81d2e2f4cd3254dd345c1e921b12bff62eb96e7551336c44fb0da5407bf5fe5f` -[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-node-linux-arm.tar.gz) | `b14734a20190aca2b2af9cee59549d285be4f0c38faf89c5308c94534110edc1` -[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-node-linux-arm64.tar.gz) | `ad0a81ecf6ef8346b7aa98a8d02a4f3853d0a5439d149a14b1ac2307b763b2ad` -[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-node-linux-ppc64le.tar.gz) | `8e6d72837fe19afd055786c8731bd555fe082e107195c956c6985e56a03d504f` -[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-node-linux-s390x.tar.gz) | `0fc7d55fb2750b29c0bbc36da050c8bf14508b1aa40e38e3b7f6cf311b464827` -[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.12.0-beta.1/kubernetes-node-windows-amd64.tar.gz) | `09bf133156b9bc474d272bf16e765b143439959a1f007283c477e7999f2b4d6a` +[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-node-linux-amd64.tar.gz) | `e3cba71c2b2d151cdcc44937c1bea083ee0ceb829e7feb25cd37edd4d0bd7adac0a166144bd33f9aa616d730785d52c3fee70f0db272d892e79c38eb544cfd22` +[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-node-linux-arm.tar.gz) | `28d7f1cf4fecdc72da7f5f19836cc06bb08182f8e8fb1641dc01e9299247905e305b04c676efa9afbdb8b1ed649542148406b110b75b1e87c1daca7ffed6018c` +[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-node-linux-arm64.tar.gz) | `cfe22b11502cd857f0e277e7c1af08e6202f7ffc36f852c6154159bdd67bb330ffd18e5f768cf6968f0524aa80ffba6b7012207831ba94dec4084f1ce38a5289` +[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-node-linux-ppc64le.tar.gz) | `195a6785c49af419361a8901c99bb6613a6578a8eac5e8f08ec28077645be18c6d04c9afc2839d07de27b15595584f6dff5e5f9e78001fc2c6a3249e99b82247` +[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-node-linux-s390x.tar.gz) | `51f5b5ed47b50f5188d9e2f57b03555492d3e490494842247fa04fe81ea350c6694a52caa46ea50578ca07e48212f4f714b8614d8b1eb8924b424eaed73a8d24` +[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.2/kubernetes-node-windows-amd64.tar.gz) | `d2690d57cd485c0c7ebe425464ad59f2c7722870abd6f264ea7fae65a4e403c89446373ba9cd23cd43e8316902637aa9e0b23d571b0ac890c35ff8e9c3ff3c3b` -## Changelog since v1.12.0-alpha.1 - -### Action Required - -* Move volume dynamic provisioning scheduling to beta (ACTION REQUIRED: The DynamicProvisioningScheduling alpha feature gate has been removed. The VolumeScheduling beta feature gate is still required for this feature) ([#67432](https://github.com/kubernetes/kubernetes/pull/67432), [@lichuqiang](https://github.com/lichuqiang)) +## Changelog since v1.13.0-alpha.1 ### Other notable changes -* Not split nodes when searching for nodes but doing it all at once. ([#67555](https://github.com/kubernetes/kubernetes/pull/67555), [@wgliang](https://github.com/wgliang)) -* Deprecate kubectl run generators, except for run-pod/v1 ([#68132](https://github.com/kubernetes/kubernetes/pull/68132), [@soltysh](https://github.com/soltysh)) -* Using the Horizontal Pod Autoscaler with metrics from Heapster is now deprecated. ([#68089](https://github.com/kubernetes/kubernetes/pull/68089), [@DirectXMan12](https://github.com/DirectXMan12)) -* Support both directory and block device for local volume plugin FileSystem VolumeMode ([#63011](https://github.com/kubernetes/kubernetes/pull/63011), [@NickrenREN](https://github.com/NickrenREN)) -* Add CSI volume attributes for kubectl describe pv. ([#65074](https://github.com/kubernetes/kubernetes/pull/65074), [@wgliang](https://github.com/wgliang)) -* `kubectl rollout status` now works for unlimited timeouts. ([#67817](https://github.com/kubernetes/kubernetes/pull/67817), [@tnozicka](https://github.com/tnozicka)) -* Fix panic when processing Azure HTTP response. ([#68210](https://github.com/kubernetes/kubernetes/pull/68210), [@feiskyer](https://github.com/feiskyer)) -* add mixed protocol support for azure load balancer ([#67986](https://github.com/kubernetes/kubernetes/pull/67986), [@andyzhangx](https://github.com/andyzhangx)) -* Replace scale down forbidden window with scale down stabilization window. Rather than waiting a fixed period of time between scale downs HPA now scales down to the highest recommendation it during the scale down stabilization window. ([#68122](https://github.com/kubernetes/kubernetes/pull/68122), [@krzysztof-jastrzebski](https://github.com/krzysztof-jastrzebski)) -* Adding validation to kube-scheduler at the API level ([#66799](https://github.com/kubernetes/kubernetes/pull/66799), [@noqcks](https://github.com/noqcks)) -* Improve performance of Pod affinity/anti-affinity in the scheduler ([#67788](https://github.com/kubernetes/kubernetes/pull/67788), [@ahmad-diaa](https://github.com/ahmad-diaa)) -* kubeadm: fix air-gapped support and also allow some kubeadm commands to work without an available networking interface ([#67397](https://github.com/kubernetes/kubernetes/pull/67397), [@neolit123](https://github.com/neolit123)) -* Increase Horizontal Pod Autoscaler default update interval (30s -> 15s). It will improve HPA reaction time for metric changes. ([#68021](https://github.com/kubernetes/kubernetes/pull/68021), [@krzysztof-jastrzebski](https://github.com/krzysztof-jastrzebski)) -* Increase scrape frequency of metrics-server to 30s ([#68127](https://github.com/kubernetes/kubernetes/pull/68127), [@serathius](https://github.com/serathius)) -* Add new `--server-dry-run` flag to `kubectl apply` so that the request will be sent to the server with the dry-run flag (alpha), which means that changes won't be persisted. ([#68069](https://github.com/kubernetes/kubernetes/pull/68069), [@apelisse](https://github.com/apelisse)) -* kubelet v1beta1 external ComponentConfig types are now available in the `k8s.io/kubelet` repo ([#67263](https://github.com/kubernetes/kubernetes/pull/67263), [@luxas](https://github.com/luxas)) -* Adds a kubelet parameter and config option to change CFS quota period from the default 100ms to some other value between 1µs and 1s. This was done to improve response latencies for workloads running in clusters with guaranteed and burstable QoS classes. ([#63437](https://github.com/kubernetes/kubernetes/pull/63437), [@szuecs](https://github.com/szuecs)) -* Enable secure serving on port 10258 to cloud-controller-manager (configurable via `--secure-port`). Delegated authentication and authorization have to be configured like for aggregated API servers. ([#67069](https://github.com/kubernetes/kubernetes/pull/67069), [@sttts](https://github.com/sttts)) -* Support extra `--prune-whitelist` resources in kube-addon-manager. ([#67743](https://github.com/kubernetes/kubernetes/pull/67743), [@Random-Liu](https://github.com/Random-Liu)) -* Upon receiving a LIST request with expired continue token, the apiserver now returns a continue token together with the 410 "the from parameter is too old " error. If the client does not care about getting a list from a consistent snapshot, the client can use this token to continue listing from the next key, but the returned chunk will be from the latest snapshot. ([#67284](https://github.com/kubernetes/kubernetes/pull/67284), [@caesarxuchao](https://github.com/caesarxuchao)) -* Role, ClusterRole and their bindings for cloud-provider is put under system namespace. Their addonmanager mode switches to EnsureExists. ([#67224](https://github.com/kubernetes/kubernetes/pull/67224), [@grayluck](https://github.com/grayluck)) -* Mount propagation has promoted to GA. The `MountPropagation` feature gate is deprecated and will be removed in 1.13. ([#67255](https://github.com/kubernetes/kubernetes/pull/67255), [@bertinatto](https://github.com/bertinatto)) -* Introduce CSI Cluster Registration mechanism to ease CSI plugin discovery and allow CSI drivers to customize Kubernetes' interaction with them. ([#67803](https://github.com/kubernetes/kubernetes/pull/67803), [@saad-ali](https://github.com/saad-ali)) -* Adds the commands `kubeadm alpha phases renew ` ([#67910](https://github.com/kubernetes/kubernetes/pull/67910), [@liztio](https://github.com/liztio)) -* ProcMount added to SecurityContext and AllowedProcMounts added to PodSecurityPolicy to allow paths in the container's /proc to not be masked. ([#64283](https://github.com/kubernetes/kubernetes/pull/64283), [@jessfraz](https://github.com/jessfraz)) -* support cross resource group for azure file ([#68117](https://github.com/kubernetes/kubernetes/pull/68117), [@andyzhangx](https://github.com/andyzhangx)) -* Port 31337 will be used by fluentd ([#68051](https://github.com/kubernetes/kubernetes/pull/68051), [@Szetty](https://github.com/Szetty)) -* Improve CPU sample sanitization in HPA by taking metric's freshness into account. ([#68068](https://github.com/kubernetes/kubernetes/pull/68068), [@krzysztof-jastrzebski](https://github.com/krzysztof-jastrzebski)) -* CoreDNS is now v1.2.2 for Kubernetes 1.12 ([#68076](https://github.com/kubernetes/kubernetes/pull/68076), [@rajansandeep](https://github.com/rajansandeep)) -* Enable secure serving on port 10257 to kube-controller-manager (configurable via `--secure-port`). Delegated authentication and authorization have to be configured like for aggregated API servers. ([#64149](https://github.com/kubernetes/kubernetes/pull/64149), [@sttts](https://github.com/sttts)) -* Update metrics-server to v0.3.0. ([#68077](https://github.com/kubernetes/kubernetes/pull/68077), [@DirectXMan12](https://github.com/DirectXMan12)) -* TokenRequest and TokenRequestProjection are now beta features. To enable these feature, the API server needs to be started with the following flags: ([#67349](https://github.com/kubernetes/kubernetes/pull/67349), [@mikedanese](https://github.com/mikedanese)) - * --service-account-issuer - * --service-account-signing-key-file - * --service-account-api-audiences -* Don't let aggregated apiservers fail to launch if the external-apiserver-authentication configmap is not found in the cluster. ([#67836](https://github.com/kubernetes/kubernetes/pull/67836), [@sttts](https://github.com/sttts)) -* Promote AdvancedAuditing to GA, replacing the previous (legacy) audit logging mechanisms. ([#65862](https://github.com/kubernetes/kubernetes/pull/65862), [@loburm](https://github.com/loburm)) -* Azure cloud provider now supports unmanaged nodes (such as on-prem) that are labeled with `kubernetes.azure.com/managed=false` and `alpha.service-controller.kubernetes.io/exclude-balancer=true` ([#67984](https://github.com/kubernetes/kubernetes/pull/67984), [@feiskyer](https://github.com/feiskyer)) -* `kubectl get apiservice` now shows the target service and whether the service is available ([#67747](https://github.com/kubernetes/kubernetes/pull/67747), [@smarterclayton](https://github.com/smarterclayton)) -* Openstack supports now node shutdown taint. Taint is added when instance is shutdown in openstack. ([#67982](https://github.com/kubernetes/kubernetes/pull/67982), [@zetaab](https://github.com/zetaab)) -* Return apiserver panics as 500 errors instead terminating the apiserver process. ([#68001](https://github.com/kubernetes/kubernetes/pull/68001), [@sttts](https://github.com/sttts)) -* Fix VMWare VM freezing bug by reverting [#51066](https://github.com/kubernetes/kubernetes/pull/51066) ([#67825](https://github.com/kubernetes/kubernetes/pull/67825), [@nikopen](https://github.com/nikopen)) -* Make CoreDNS be the default DNS server in kube-up (instead of kube-dns formerly). ([#67569](https://github.com/kubernetes/kubernetes/pull/67569), [@fturib](https://github.com/fturib)) - * It is still possible to deploy kube-dns by setting CLUSTER_DNS_CORE_DNS=false. -* Added support to restore a volume from a volume snapshot data source. ([#67087](https://github.com/kubernetes/kubernetes/pull/67087), [@xing-yang](https://github.com/xing-yang)) -* fixes the errors/warnings in fluentd configuration ([#67947](https://github.com/kubernetes/kubernetes/pull/67947), [@saravanan30erd](https://github.com/saravanan30erd)) -* Stop counting soft-deleted pods for scaling purposes in HPA controller to avoid soft-deleted pods incorrectly affecting scale up replica count calculation. ([#67067](https://github.com/kubernetes/kubernetes/pull/67067), [@moonek](https://github.com/moonek)) -* delegated authn/z: optionally opt-out of mandatory authn/authz kubeconfig ([#67545](https://github.com/kubernetes/kubernetes/pull/67545), [@sttts](https://github.com/sttts)) -* kubeadm: Control plane images (etcd, kube-apiserver, kube-proxy, etc.) don't use arch suffixes. Arch suffixes are kept for kube-dns only. ([#66960](https://github.com/kubernetes/kubernetes/pull/66960), [@rosti](https://github.com/rosti)) -* Adds sample-cli-plugin staging repository ([#67938](https://github.com/kubernetes/kubernetes/pull/67938), [@soltysh](https://github.com/soltysh)) -* adjusted http/2 buffer sizes for apiservers to prevent starvation issues between concurrent streams ([#67902](https://github.com/kubernetes/kubernetes/pull/67902), [@liggitt](https://github.com/liggitt)) -* SCTP is now supported as additional protocol (alpha) alongside TCP and UDP in Pod, Service, Endpoint, and NetworkPolicy. ([#64973](https://github.com/kubernetes/kubernetes/pull/64973), [@janosi](https://github.com/janosi)) -* Always create configmaps/extensions-apiserver-authentication from kube-apiserver. ([#67694](https://github.com/kubernetes/kubernetes/pull/67694), [@sttts](https://github.com/sttts)) -* kube-proxy v1beta1 external ComponentConfig types are now available in the `k8s.io/kube-proxy` repo ([#67688](https://github.com/kubernetes/kubernetes/pull/67688), [@Lion-Wei](https://github.com/Lion-Wei)) -* Apply unreachable taint to a node when it lost network connection. ([#67734](https://github.com/kubernetes/kubernetes/pull/67734), [@Huang-Wei](https://github.com/Huang-Wei)) -* Allow ImageReview backend to return annotations to be added to the created pod. ([#64597](https://github.com/kubernetes/kubernetes/pull/64597), [@wteiken](https://github.com/wteiken)) -* Bump ip-masq-agent to v2.1.1 ([#67916](https://github.com/kubernetes/kubernetes/pull/67916), [@MrHohn](https://github.com/MrHohn)) - * - Update debian-iptables image for CVEs. - * - Change chain name to IP-MASQ to be compatible with the - * pre-injected masquerade rules. -* AllowedTopologies field inside StorageClass is now validated against set and map semantics. Specifically, there cannot be duplicate TopologySelectorTerms, MatchLabelExpressions keys, and TopologySelectorLabelRequirement Values. ([#66843](https://github.com/kubernetes/kubernetes/pull/66843), [@verult](https://github.com/verult)) -* Introduces autoscaling/v2beta2 and custom_metrics/v1beta2, which implement metric selectors for Object and Pods metrics, as well as allowing AverageValue targets on Objects, similar to External metrics. ([#64097](https://github.com/kubernetes/kubernetes/pull/64097), [@damemi](https://github.com/damemi)) -* The cloudstack cloud provider now reports a `Hostname` address type for nodes based on the `local-hostname` metadata key. ([#67719](https://github.com/kubernetes/kubernetes/pull/67719), [@liggitt](https://github.com/liggitt)) -* kubeadm: --cri-socket now defaults to tcp://localhost:2375 when running on Windows ([#67447](https://github.com/kubernetes/kubernetes/pull/67447), [@benmoss](https://github.com/benmoss)) -* kubeadm: The kubeadm configuration now support definition of more than one control plane instances with their own APIEndpoint. The APIEndpoint for the "bootstrap" control plane instance should be defined using `InitConfiguration.APIEndpoint`, while the APIEndpoints for additional control plane instances should be added using `JoinConfiguration.APIEndpoint`. ([#67832](https://github.com/kubernetes/kubernetes/pull/67832), [@fabriziopandini](https://github.com/fabriziopandini)) -* Enable dynamic azure disk volume limits ([#67772](https://github.com/kubernetes/kubernetes/pull/67772), [@andyzhangx](https://github.com/andyzhangx)) -* kubelet: Users can now enable the alpha NodeLease feature gate to have the Kubelet create and periodically renew a Lease in the kube-node-lease namespace. The lease duration defaults to 40s, and can be configured via the kubelet.config.k8s.io/v1beta1.KubeletConfiguration's NodeLeaseDurationSeconds field. ([#66257](https://github.com/kubernetes/kubernetes/pull/66257), [@mtaufen](https://github.com/mtaufen)) -* latent controller caches no longer cause repeating deletion messages for deleted pods ([#67826](https://github.com/kubernetes/kubernetes/pull/67826), [@deads2k](https://github.com/deads2k)) -* API paging is now enabled for custom resource definitions, custom resources and APIService objects ([#67861](https://github.com/kubernetes/kubernetes/pull/67861), [@liggitt](https://github.com/liggitt)) -* kubeadm: ControlPlaneEndpoint was moved from the API config struct to ClusterConfiguration ([#67830](https://github.com/kubernetes/kubernetes/pull/67830), [@fabriziopandini](https://github.com/fabriziopandini)) -* kubeadm - feature-gates HighAvailability, SelfHosting, CertsInSecrets are now deprecated and can't be used anymore for new clusters. Update of cluster using above feature-gates flag is not supported ([#67786](https://github.com/kubernetes/kubernetes/pull/67786), [@fabriziopandini](https://github.com/fabriziopandini)) -* Replace scale up forbidden window with disregarding CPU samples collected when pod was initializing. ([#67252](https://github.com/kubernetes/kubernetes/pull/67252), [@jbartosik](https://github.com/jbartosik)) -* Moving KubeSchedulerConfiguration from ComponentConfig API types to staging repos ([#66916](https://github.com/kubernetes/kubernetes/pull/66916), [@dixudx](https://github.com/dixudx)) -* Improved error message when checking the rollout status of StatefulSet with OnDelete strategy type ([#66983](https://github.com/kubernetes/kubernetes/pull/66983), [@mortent](https://github.com/mortent)) -* RuntimeClass is a new API resource for defining different classes of runtimes that may be used to run containers in the cluster. Pods can select a RunitmeClass to use via the RuntimeClassName field. This feature is in alpha, and the RuntimeClass feature gate must be enabled in order to use it. ([#67737](https://github.com/kubernetes/kubernetes/pull/67737), [@tallclair](https://github.com/tallclair)) -* Remove rescheduler since scheduling DS pods by default scheduler is moving to beta. ([#67687](https://github.com/kubernetes/kubernetes/pull/67687), [@Lion-Wei](https://github.com/Lion-Wei)) -* Turn on PodReadinessGate by default ([#67406](https://github.com/kubernetes/kubernetes/pull/67406), [@freehan](https://github.com/freehan)) -* Speed up kubelet start time by executing an immediate runtime and node status update when the Kubelet sees that it has a CIDR. ([#67031](https://github.com/kubernetes/kubernetes/pull/67031), [@krzysztof-jastrzebski](https://github.com/krzysztof-jastrzebski)) -* The OpenStack cloud provider now reports a `Hostname` address type for nodes ([#67748](https://github.com/kubernetes/kubernetes/pull/67748), [@FengyunPan2](https://github.com/FengyunPan2)) -* The aws cloud provider now reports a `Hostname` address type for nodes based on the `local-hostname` metadata key. ([#67715](https://github.com/kubernetes/kubernetes/pull/67715), [@liggitt](https://github.com/liggitt)) -* Azure cloud provider now supports cross resource group nodes that are labeled with `kubernetes.azure.com/resource-group=` and `alpha.service-controller.kubernetes.io/exclude-balancer=true` ([#67604](https://github.com/kubernetes/kubernetes/pull/67604), [@feiskyer](https://github.com/feiskyer)) -* Reduce API calls for Azure instance metadata. ([#67478](https://github.com/kubernetes/kubernetes/pull/67478), [@feiskyer](https://github.com/feiskyer)) -* `kubectl create secret tls` can now read certificate and key files from process substitution arguments ([#67713](https://github.com/kubernetes/kubernetes/pull/67713), [@liggitt](https://github.com/liggitt)) -* change default value of kind for azure disk ([#67483](https://github.com/kubernetes/kubernetes/pull/67483), [@andyzhangx](https://github.com/andyzhangx)) -* To address the possibility dry-run requests overwhelming admission webhooks that rely on side effects and a reconciliation mechanism, a new field is being added to admissionregistration.k8s.io/v1beta1.ValidatingWebhookConfiguration and admissionregistration.k8s.io/v1beta1.MutatingWebhookConfiguration so that webhooks can explicitly register as having dry-run support. If a dry-run request is made on a resource that triggers a non dry-run supporting webhook, the request will be completely rejected, with "400: Bad Request". Additionally, a new field is being added to the admission.k8s.io/v1beta1.AdmissionReview API object, exposing to webhooks whether or not the request being reviewed is a dry-run. ([#66936](https://github.com/kubernetes/kubernetes/pull/66936), [@jennybuckley](https://github.com/jennybuckley)) -* Kubeadm ha upgrade ([#66973](https://github.com/kubernetes/kubernetes/pull/66973), [@fabriziopandini](https://github.com/fabriziopandini)) -* kubeadm: InitConfiguration now consists of two structs: InitConfiguration and ClusterConfiguration ([#67441](https://github.com/kubernetes/kubernetes/pull/67441), [@rosti](https://github.com/rosti)) -* Updated Cluster Autoscaler version to 1.3.2-beta.2. Release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.3.2-beta.2 ([#67697](https://github.com/kubernetes/kubernetes/pull/67697), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) -* cpumanager: rollback state if updateContainerCPUSet failed ([#67430](https://github.com/kubernetes/kubernetes/pull/67430), [@choury](https://github.com/choury)) -* [CRI] Adds a "runtime_handler" field to RunPodSandboxRequest, for selecting the runtime configuration to run the sandbox with (alpha feature). ([#67518](https://github.com/kubernetes/kubernetes/pull/67518), [@tallclair](https://github.com/tallclair)) -* Create cli-runtime staging repository ([#67658](https://github.com/kubernetes/kubernetes/pull/67658), [@soltysh](https://github.com/soltysh)) -* Headless Services with no ports defined will now create Endpoints correctly, and appear in DNS. ([#67622](https://github.com/kubernetes/kubernetes/pull/67622), [@thockin](https://github.com/thockin)) -* Kubernetes juju charms will now use CSI for ceph. ([#66523](https://github.com/kubernetes/kubernetes/pull/66523), [@hyperbolic2346](https://github.com/hyperbolic2346)) -* kubeadm: Fix panic when node annotation is nil ([#67648](https://github.com/kubernetes/kubernetes/pull/67648), [@xlgao-zju](https://github.com/xlgao-zju)) -* Prevent `resourceVersion` updates for custom resources on no-op writes. ([#67562](https://github.com/kubernetes/kubernetes/pull/67562), [@nikhita](https://github.com/nikhita)) -* Fail container start if its requested device plugin resource hasn't registered after Kubelet restart. ([#67145](https://github.com/kubernetes/kubernetes/pull/67145), [@jiayingz](https://github.com/jiayingz)) -* Use sync.map to scale ecache better ([#66862](https://github.com/kubernetes/kubernetes/pull/66862), [@resouer](https://github.com/resouer)) -* DaemonSet: Fix bug- daemonset didn't create pod after node have enough resource ([#67337](https://github.com/kubernetes/kubernetes/pull/67337), [@linyouchong](https://github.com/linyouchong)) -* updates kibana to 6.3.2 ([#67582](https://github.com/kubernetes/kubernetes/pull/67582), [@monotek](https://github.com/monotek)) -* fixes json logging in fluentd-elasticsearch image by downgrading fluent-plugin-kubernetes_metadata_filter plugin to version 2.0.0 ([#67544](https://github.com/kubernetes/kubernetes/pull/67544), [@monotek](https://github.com/monotek)) -* add --dns-loop-detect option to dnsmasq run by kube-dns ([#67302](https://github.com/kubernetes/kubernetes/pull/67302), [@dixudx](https://github.com/dixudx)) -* Switched certificate data replacement from "REDACTED" to "DATA+OMITTED" ([#66023](https://github.com/kubernetes/kubernetes/pull/66023), [@ibrasho](https://github.com/ibrasho)) -* improve performance of anti-affinity predicate of default scheduler. ([#66948](https://github.com/kubernetes/kubernetes/pull/66948), [@mohamed-mehany](https://github.com/mohamed-mehany)) -* Fixed a bug that was blocking extensible error handling when serializing API responses error out. Previously, serialization failures always resulted in the status code of the original response being returned. Now, the following behavior occurs: ([#67041](https://github.com/kubernetes/kubernetes/pull/67041), [@tristanburgess](https://github.com/tristanburgess)) - * - If the serialization type is application/vnd.kubernetes.protobuf, and protobuf marshaling is not implemented for the requested API resource type, a '406 Not Acceptable is returned'. - * - If the serialization type is 'application/json': - * - If serialization fails, and the original status code was an failure (e.g. 4xx or 5xx), the original status code will be returned. - * - If serialization fails, and the original status code was not a failure (e.g. 2xx), the status code of the serialization failure will be returned. By default, this is '500 Internal Server Error', because JSON serialization is our default, and not supposed to be implemented on a type-by-type basis. -* Add a feature to the scheduler to score fewer than all nodes in every scheduling cycle. This can improve performance of the scheduler in large clusters. ([#66733](https://github.com/kubernetes/kubernetes/pull/66733), [@bsalamat](https://github.com/bsalamat)) -* kube-controller-manager can now start the quota controller when discovery results can only be partially determined. ([#67433](https://github.com/kubernetes/kubernetes/pull/67433), [@deads2k](https://github.com/deads2k)) -* The plugin mechanism functionality now closely follows the git plugin design ([#66876](https://github.com/kubernetes/kubernetes/pull/66876), [@juanvallejo](https://github.com/juanvallejo)) -* GCE: decrease cpu requests on master node, to allow more components to fit on one core machine. ([#67504](https://github.com/kubernetes/kubernetes/pull/67504), [@loburm](https://github.com/loburm)) -* PVC may not be synced to controller local cache in time if PV is bound by external PV binder (e.g. kube-scheduler), double check if PVC is not found to prevent reclaiming PV wrongly. ([#67062](https://github.com/kubernetes/kubernetes/pull/67062), [@cofyc](https://github.com/cofyc)) -* add more storage account sku support for azure disk ([#67528](https://github.com/kubernetes/kubernetes/pull/67528), [@andyzhangx](https://github.com/andyzhangx)) -* updates es-image to elasticsearch 6.3.2 ([#67484](https://github.com/kubernetes/kubernetes/pull/67484), [@monotek](https://github.com/monotek)) -* Bump GLBC version to 1.2.3 ([#66793](https://github.com/kubernetes/kubernetes/pull/66793), [@freehan](https://github.com/freehan)) -* kube-apiserver: fixes error creating system priority classes when starting multiple apiservers simultaneously ([#67372](https://github.com/kubernetes/kubernetes/pull/67372), [@tanshanshan](https://github.com/tanshanshan)) -* kubectl patch now respects --local ([#67399](https://github.com/kubernetes/kubernetes/pull/67399), [@deads2k](https://github.com/deads2k)) -* Defaults for file audit logging backend in batch mode changed: ([#67223](https://github.com/kubernetes/kubernetes/pull/67223), [@tallclair](https://github.com/tallclair)) - * - Logs are written 1 at a time (no batching) - * - Only a single writer process (lock contention) -* Forget rate limit when CRD establish controller successfully updated CRD condition ([#67370](https://github.com/kubernetes/kubernetes/pull/67370), [@yue9944882](https://github.com/yue9944882)) -* updates fluentd in fluentd-elasticsearch to version 1.2.4 ([#67434](https://github.com/kubernetes/kubernetes/pull/67434), [@monotek](https://github.com/monotek)) - * also updates activesupport, fluent-plugin-elasticsearch & oj gems -* The dockershim now sets the "bandwidth" and "ipRanges" CNI capabilities (dynamic parameters). Plugin authors and administrators can now take advantage of this by updating their CNI configuration file. For more information, see the [CNI docs](https://github.com/containernetworking/cni/blob/master/CONVENTIONS.md#dynamic-plugin-specific-fields-capabilities--runtime-configuration) ([#64445](https://github.com/kubernetes/kubernetes/pull/64445), [@squeed](https://github.com/squeed)) -* Expose `/debug/flags/v` to allow kubelet dynamically set glog logging level. If want to change glog level to 3, you only have to send a PUT request like `curl -X PUT http://127.0.0.1:8080/debug/flags/v -d "3"`. ([#64601](https://github.com/kubernetes/kubernetes/pull/64601), [@hzxuzhonghu](https://github.com/hzxuzhonghu)) -* Fix an issue that pods using hostNetwork keep increasing. ([#67456](https://github.com/kubernetes/kubernetes/pull/67456), [@Huang-Wei](https://github.com/Huang-Wei)) -* DaemonSet controller is now using backoff algorithm to avoid hot loops fighting with kubelet on pod recreation when a particular DaemonSet is misconfigured. ([#65309](https://github.com/kubernetes/kubernetes/pull/65309), [@tnozicka](https://github.com/tnozicka)) -* Add node affinity for Azure unzoned managed disks ([#67229](https://github.com/kubernetes/kubernetes/pull/67229), [@feiskyer](https://github.com/feiskyer)) -* Attacher/Detacher refactor for local storage ([#66884](https://github.com/kubernetes/kubernetes/pull/66884), [@NickrenREN](https://github.com/NickrenREN)) -* Update debian-iptables and hyperkube-base images to include CVE fixes. ([#67365](https://github.com/kubernetes/kubernetes/pull/67365), [@ixdy](https://github.com/ixdy)) -* Fix an issue where filesystems are not unmounted when a backend is not reachable and returns EIO. ([#67097](https://github.com/kubernetes/kubernetes/pull/67097), [@chakri-nelluri](https://github.com/chakri-nelluri)) -* Update Cluster Autoscaler version to 1.3.2-beta.1. Release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.3.2-beta.1 ([#67396](https://github.com/kubernetes/kubernetes/pull/67396), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) -* Remove unused binary and container image for kube-aggregator. The functionality is already integrated into the kube-apiserver. ([#67157](https://github.com/kubernetes/kubernetes/pull/67157), [@dims](https://github.com/dims)) -* Avoid creating new controller revisions for statefulsets when cache is stale ([#67039](https://github.com/kubernetes/kubernetes/pull/67039), [@mortent](https://github.com/mortent)) -* Revert [#63905](https://github.com/kubernetes/kubernetes/pull/63905): Setup dns servers and search domains for Windows Pods. DNS for Windows containers will be set by CNI plugins. ([#66587](https://github.com/kubernetes/kubernetes/pull/66587), [@feiskyer](https://github.com/feiskyer)) -* attachdetach controller attaches volumes immediately when Pod's PVCs are bound ([#66863](https://github.com/kubernetes/kubernetes/pull/66863), [@cofyc](https://github.com/cofyc)) -* The check for unsupported plugins during volume resize has been moved from the admission controller to the two controllers that handle volume resize. ([#66780](https://github.com/kubernetes/kubernetes/pull/66780), [@kangarlou](https://github.com/kangarlou)) -* Fix kubelet to not leak goroutines/intofiy watchers on an inactive connection if it's closed ([#67285](https://github.com/kubernetes/kubernetes/pull/67285), [@yujuhong](https://github.com/yujuhong)) -* fix azure disk create failure due to sdk upgrade ([#67236](https://github.com/kubernetes/kubernetes/pull/67236), [@andyzhangx](https://github.com/andyzhangx)) -* Kubeadm join --control-plane main workflow ([#66873](https://github.com/kubernetes/kubernetes/pull/66873), [@fabriziopandini](https://github.com/fabriziopandini)) -* Dynamic provisions that create iSCSI PVs can ensure that multipath is used by specifying 2 or more target portals in the PV, which will cause kubelet to wait up to 10 seconds for the multipath device. PVs with just one portal continue to work as before, with kubelet not waiting for the multipath device and just using the first disk it finds. ([#67140](https://github.com/kubernetes/kubernetes/pull/67140), [@bswartz](https://github.com/bswartz)) -* kubectl: recreating resources for immutable fields when force is applied ([#66602](https://github.com/kubernetes/kubernetes/pull/66602), [@dixudx](https://github.com/dixudx)) -* Remove deprecated --interactive flag from kubectl logs. ([#65420](https://github.com/kubernetes/kubernetes/pull/65420), [@jsoref](https://github.com/jsoref)) -* kubeadm uses audit policy v1 instead of v1beta1 ([#67176](https://github.com/kubernetes/kubernetes/pull/67176), [@charrywanganthony](https://github.com/charrywanganthony)) -* kubeadm: make sure pre-pulled kube-proxy image and the one specified in its daemon set manifest are the same ([#67131](https://github.com/kubernetes/kubernetes/pull/67131), [@rosti](https://github.com/rosti)) -* Graduate Resource Quota ScopeSelectors to beta, and enable it by default. ([#67077](https://github.com/kubernetes/kubernetes/pull/67077), [@vikaschoudhary16](https://github.com/vikaschoudhary16)) -* Decrease the amount of time it takes to modify kubeconfig files with large amounts of contexts ([#67093](https://github.com/kubernetes/kubernetes/pull/67093), [@juanvallejo](https://github.com/juanvallejo)) -* Fixes issue when updating a DaemonSet causes a hash collision. ([#66476](https://github.com/kubernetes/kubernetes/pull/66476), [@mortent](https://github.com/mortent)) -* fix cluster-info dump error ([#66652](https://github.com/kubernetes/kubernetes/pull/66652), [@charrywanganthony](https://github.com/charrywanganthony)) -* The PodShareProcessNamespace feature to configure PID namespace sharing within a pod has been promoted to beta. ([#66507](https://github.com/kubernetes/kubernetes/pull/66507), [@verb](https://github.com/verb)) -* `kubectl create {clusterrole,role}`'s `--resources` flag supports asterisk to specify all resources. ([#62945](https://github.com/kubernetes/kubernetes/pull/62945), [@nak3](https://github.com/nak3)) -* Bump up version number of debian-base, debian-hyperkube-base and debian-iptables. ([#67026](https://github.com/kubernetes/kubernetes/pull/67026), [@satyasm](https://github.com/satyasm)) - * Also updates dependencies of users of debian-base. - * debian-base version 0.3.1 is already available. -* DynamicProvisioningScheduling and VolumeScheduling is now supported for Azure managed disks. Feature gates DynamicProvisioningScheduling and VolumeScheduling should be enabled before using this feature. ([#67121](https://github.com/kubernetes/kubernetes/pull/67121), [@feiskyer](https://github.com/feiskyer)) -* kube-apiserver now includes all registered API groups in discovery, including registered extension API group/versions for unavailable extension API servers. ([#66932](https://github.com/kubernetes/kubernetes/pull/66932), [@nilebox](https://github.com/nilebox)) -* Allows extension API server to dynamically discover the requestheader CA certificate when the core API server doesn't use certificate based authentication for it's clients ([#66394](https://github.com/kubernetes/kubernetes/pull/66394), [@rtripat](https://github.com/rtripat)) -* audit.k8s.io api group is upgraded from v1beta1 to v1. ([#65891](https://github.com/kubernetes/kubernetes/pull/65891), [@CaoShuFeng](https://github.com/CaoShuFeng)) - * Deprecated element metav1.ObjectMeta and Timestamp are removed from audit Events in v1 version. - * Default value of option --audit-webhook-version and --audit-log-version will be changed from `audit.k8s.io/v1beta1` to `audit.k8s.io/v1` in release 1.13 -* scope AWS LoadBalancer security group ICMP rules to spec.loadBalancerSourceRanges ([#63572](https://github.com/kubernetes/kubernetes/pull/63572), [@haz-mat](https://github.com/haz-mat)) -* Add NoSchedule/NoExecute tolerations to ip-masq-agent, ensuring it to be scheduled in all nodes except master. ([#66260](https://github.com/kubernetes/kubernetes/pull/66260), [@tanshanshan](https://github.com/tanshanshan)) -* The flag `--skip-preflight-checks` of kubeadm has been removed. Please use `--ignore-preflight-errors` instead. ([#62727](https://github.com/kubernetes/kubernetes/pull/62727), [@xiangpengzhao](https://github.com/xiangpengzhao)) -* The watch API endpoints prefixed with `/watch` are deprecated and will be removed in a future release. These standard method for watching resources (supported since v1.0) is to use the list API endpoints with a `?watch=true` parameter. All client-go clients have used the parameter method since v1.6.0. ([#65147](https://github.com/kubernetes/kubernetes/pull/65147), [@liggitt](https://github.com/liggitt)) -* Bump Heapster to v1.6.0-beta.1 ([#67074](https://github.com/kubernetes/kubernetes/pull/67074), [@kawych](https://github.com/kawych)) -* kube-apiserver: setting a `dryRun` query parameter on a CONNECT request will now cause the request to be rejected, consistent with behavior of other mutating API requests. Examples of CONNECT APIs are the `nodes/proxy`, `services/proxy`, `pods/proxy`, `pods/exec`, and `pods/attach` subresources. Note that this prevents sending a `dryRun` parameter to backends via `{nodes,services,pods}/proxy` subresources. ([#66083](https://github.com/kubernetes/kubernetes/pull/66083), [@jennybuckley](https://github.com/jennybuckley)) -* In clusters where the DryRun feature is enabled, dry-run requests will go through the normal admission chain. Because of this, ImagePolicyWebhook authors should especially make sure that their webhooks do not rely on side effects. ([#66391](https://github.com/kubernetes/kubernetes/pull/66391), [@jennybuckley](https://github.com/jennybuckley)) -* Metadata Agent Improvements ([#66485](https://github.com/kubernetes/kubernetes/pull/66485), [@bmoyles0117](https://github.com/bmoyles0117)) - * Bump metadata agent version to 0.2-0.0.21-1. - * Expand the metadata agent's access to all API groups. - * Remove metadata agent config maps in favor of command line flags. - * Update the metadata agent's liveness probe to a new /healthz handler. - * Logging Agent Improvements - * Bump logging agent version to 0.2-1.5.33-1-k8s-1. - * Appropriately set log severity for k8s_container. - * Fix detect exceptions plugin to analyze message field instead of log field. - * Fix detect exceptions plugin to analyze streams based on local resource id. - * Disable the metadata agent for monitored resource construction in logging. - * Disable timestamp adjustment in logs to optimize performance. - * Reduce logging agent buffer chunk limit to 512k to optimize performance. -* kubectl: the wait command now prints an error message and exits with the code 1, if there is no resources matching selectors ([#66692](https://github.com/kubernetes/kubernetes/pull/66692), [@m1kola](https://github.com/m1kola)) -* Quota admission configuration api graduated to v1beta1 ([#66156](https://github.com/kubernetes/kubernetes/pull/66156), [@vikaschoudhary16](https://github.com/vikaschoudhary16)) -* Unit tests for scopes and scope selectors in the quota spec ([#66351](https://github.com/kubernetes/kubernetes/pull/66351), [@vikaschoudhary16](https://github.com/vikaschoudhary16)) -* Print kube-apiserver --help flag help in sections. ([#64517](https://github.com/kubernetes/kubernetes/pull/64517), [@sttts](https://github.com/sttts)) -* Azure managed disks now support availability zones and new parameters `zoned`, `zone` and `zones` are added for AzureDisk storage class. ([#66553](https://github.com/kubernetes/kubernetes/pull/66553), [@feiskyer](https://github.com/feiskyer)) -* nodes: improve handling of erroneous host names ([#64815](https://github.com/kubernetes/kubernetes/pull/64815), [@dixudx](https://github.com/dixudx)) -* remove deprecated shorthand flag `-c` from `kubectl version (--client)` ([#66817](https://github.com/kubernetes/kubernetes/pull/66817), [@charrywanganthony](https://github.com/charrywanganthony)) -* Added etcd_object_count metrics for CustomResources. ([#65983](https://github.com/kubernetes/kubernetes/pull/65983), [@sttts](https://github.com/sttts)) -* Handle newlines for `command`, `args`, `env`, and `annotations` in `kubectl describe` wrapping ([#66841](https://github.com/kubernetes/kubernetes/pull/66841), [@smarterclayton](https://github.com/smarterclayton)) -* Fix pod launch by kubelet when --cgroups-per-qos=false and --cgroup-driver="systemd" ([#66617](https://github.com/kubernetes/kubernetes/pull/66617), [@pravisankar](https://github.com/pravisankar)) -* kubelet: fix nil pointer dereference while enforce-node-allocatable flag is not config properly ([#66190](https://github.com/kubernetes/kubernetes/pull/66190), [@linyouchong](https://github.com/linyouchong)) -* Fix a bug on GCE that /etc/crictl.yaml is not generated when crictl is preloaded. ([#66877](https://github.com/kubernetes/kubernetes/pull/66877), [@Random-Liu](https://github.com/Random-Liu)) -* This fix prevents a GCE PD volume from being mounted if the udev device link is stale and tries to correct the link. ([#66832](https://github.com/kubernetes/kubernetes/pull/66832), [@msau42](https://github.com/msau42)) +* Corrected family type (inet6) for ipsets in ipv6-only clusters ([#68436](https://github.com/kubernetes/kubernetes/pull/68436), [@uablrek](https://github.com/uablrek)) +* Corrects check for non-Azure managed nodes with the Azure cloud provider ([#70135](https://github.com/kubernetes/kubernetes/pull/70135), [@marc-sensenich](https://github.com/marc-sensenich)) +* Windows runtime endpoints is now switched to 'npipe:////./pipe/dockershim' from 'tcp://localhost:3735'. ([#69516](https://github.com/kubernetes/kubernetes/pull/69516), [@feiskyer](https://github.com/feiskyer)) +* The caBundle and service fields in admission webhook API objects now correctly indicate they are optional ([#70138](https://github.com/kubernetes/kubernetes/pull/70138), [@liggitt](https://github.com/liggitt)) +* The --service-account-api-audiences on kube-apiserver is deprecated in favor of --api-audiences. ([#70105](https://github.com/kubernetes/kubernetes/pull/70105), [@mikedanese](https://github.com/mikedanese)) +* kubeadm: fix unnecessary upgrades caused by undefined order of Volumes and VolumeMounts in manifests ([#70027](https://github.com/kubernetes/kubernetes/pull/70027), [@bart0sh](https://github.com/bart0sh)) +* kubeadm: Implemented preflight check to ensure that number of CPUs ([#70048](https://github.com/kubernetes/kubernetes/pull/70048), [@bart0sh](https://github.com/bart0sh)) + * on the master node is not less than required. +* Reduce memory utilization of admission webhook metrics by removing resource related labels. ([#69895](https://github.com/kubernetes/kubernetes/pull/69895), [@jpbetz](https://github.com/jpbetz)) +* kubeadm: Introduce config print init/join-defaults that deprecate config print-defaults by decoupling init and join configs. ([#69617](https://github.com/kubernetes/kubernetes/pull/69617), [@rosti](https://github.com/rosti)) +* Images based on debian-base no longer include the libsystemd0 package. This should have no user-facing impact. ([#69995](https://github.com/kubernetes/kubernetes/pull/69995), [@ixdy](https://github.com/ixdy)) + * Additionally, the addon-manager image is updated to use kubectl v1.11.3. +* fix 'kubeadm upgrade' infinite loop waiting for pod restart ([#69886](https://github.com/kubernetes/kubernetes/pull/69886), [@bart0sh](https://github.com/bart0sh)) +* add more logging for azure disk diagnostics ([#70012](https://github.com/kubernetes/kubernetes/pull/70012), [@andyzhangx](https://github.com/andyzhangx)) +* Fluentd: concatenate long logs ([#68012](https://github.com/kubernetes/kubernetes/pull/68012), [@desaintmartin](https://github.com/desaintmartin)) +* CoreDNS is now the default DNS server in kube-up deployments. ([#69883](https://github.com/kubernetes/kubernetes/pull/69883), [@chrisohaver](https://github.com/chrisohaver)) +* Optimizes calculating stats when only CPU and Memory stats are returned from Kubelet stats/summary http endpoint. ([#68841](https://github.com/kubernetes/kubernetes/pull/68841), [@krzysztof-jastrzebski](https://github.com/krzysztof-jastrzebski)) +* kubeadm: Fix node join taints. ([#69846](https://github.com/kubernetes/kubernetes/pull/69846), [@andrewrynhard](https://github.com/andrewrynhard)) +* Opt out of chowning and chmoding from kubectl cp. ([#69573](https://github.com/kubernetes/kubernetes/pull/69573), [@bjhaid](https://github.com/bjhaid)) +* support Azure premium file for azure file plugin ([#69718](https://github.com/kubernetes/kubernetes/pull/69718), [@andyzhangx](https://github.com/andyzhangx)) +* `TaintBasedEvictions` feature is promoted to beta. ([#69824](https://github.com/kubernetes/kubernetes/pull/69824), [@Huang-Wei](https://github.com/Huang-Wei)) +* improves memory use and performance when processing large numbers of pods containing tolerations ([#65350](https://github.com/kubernetes/kubernetes/pull/65350), [@liggitt](https://github.com/liggitt)) +* Add dynamic audit configuration api ([#67547](https://github.com/kubernetes/kubernetes/pull/67547), [@pbarker](https://github.com/pbarker)) +* Promote resource limits priority function to beta ([#69437](https://github.com/kubernetes/kubernetes/pull/69437), [@ravisantoshgudimetla](https://github.com/ravisantoshgudimetla)) +* Fix cluster autoscaler addon permissions so it can access batch/job. ([#69858](https://github.com/kubernetes/kubernetes/pull/69858), [@losipiuk](https://github.com/losipiuk)) +* change default azure file mount permission to 0777 ([#69854](https://github.com/kubernetes/kubernetes/pull/69854), [@andyzhangx](https://github.com/andyzhangx)) +* kubeadm: JoinConfiguration now houses the discovery options in a nested Discovery structure, which in turn has a couple of other nested structures to house more specific options (BootstrapTokenDiscovery and FileDiscovery) ([#67763](https://github.com/kubernetes/kubernetes/pull/67763), [@rosti](https://github.com/rosti)) +* Fix tests to use fsync instead of sync ([#69755](https://github.com/kubernetes/kubernetes/pull/69755), [@mrunalp](https://github.com/mrunalp)) +* kube-proxy argument `hostname-override` can be used to override hostname defined in the configuration file ([#69340](https://github.com/kubernetes/kubernetes/pull/69340), [@stevesloka](https://github.com/stevesloka)) +* kube-apiserver: the `--deserialization-cache-size` flag is no longer used, is deprecated, and will be removed in a future release ([#69842](https://github.com/kubernetes/kubernetes/pull/69842), [@liggitt](https://github.com/liggitt)) +* Add support for JSON patch in fake client ([#69330](https://github.com/kubernetes/kubernetes/pull/69330), [@vaikas-google](https://github.com/vaikas-google)) -# v1.12.0-alpha.1 +# v1.13.0-alpha.1 -[Documentation](https://docs.k8s.io) & [Examples](https://releases.k8s.io/master/examples) +[Documentation](https://docs.k8s.io) -## Downloads for v1.12.0-alpha.1 +## Downloads for v1.13.0-alpha.1 -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes.tar.gz) | `603345769f5e2306e5c22db928aa1cbedc6af63f387ab7a8818cb0111292133f` -[kubernetes-src.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-src.tar.gz) | `f8fb4610cee20195381e54bfd163fbaeae228d68986817b685948b8957f324d0` +[kubernetes.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes.tar.gz) | `9f8a34b54a22ea4d7925c2f8d0e0cb2e2005486b1ed89e594bc0100ec7202fc247b89c5cbde5dc50c1f9d9f27e4f92aa0ca71fdffb9d079f63751bb1859d5bb4` +[kubernetes-src.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-src.tar.gz) | `a27a7c254d3677c823bd6fd1d0d5f9b1e78ccf807837173669a0079b0812a23444d646d80c2433c167ae50bf1a0e2a4b1d7cbd7457a505fc666464b069bd1e5f` ### Client Binaries -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-client-darwin-386.tar.gz) | `e081c275601bcaa45d906a976d35902256f836bb60caa738a2fd8719ff3e1048` -[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-client-darwin-amd64.tar.gz) | `2dd222a267ac247dce4dfc52aff313f20c427b4351f7410aadebe8569ede3139` -[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-client-linux-386.tar.gz) | `46b16d6b0429163da67b06242772c3c6c5ab9da6deda5306e63d21be04b4811d` -[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-client-linux-amd64.tar.gz) | `8b8bf0a8a4568559d3762a72c1095ab37785fc8bbbb290aaff3a34341a24d7eb` -[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-client-linux-arm.tar.gz) | `d71dc60e087746b2832e66170053816dc8ed42e95efe0769ed926a6e044175ef` -[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-client-linux-arm64.tar.gz) | `e9091bbfb997d1603dfd17ba9f145ca7dacf304f04d10230e056f8a12ce44445` -[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-client-linux-ppc64le.tar.gz) | `fc6c0985ccbd806add497f2557000f7e90f3176427250e019a40e8acf7c42282` -[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-client-linux-s390x.tar.gz) | `b8c64b318d702f6e8be76330fd5da9b87e2e4e31e904ea7e00c0cd6412ab2bcf` -[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-client-windows-386.tar.gz) | `cb96e353eb5d400756a93c8d16321d0fac87d6a4f8ad89fda42858f8e4d85e9d` -[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-client-windows-amd64.tar.gz) | `003284f983cafc6fd0ce1205c03d47e638a999def1ef4e1e77bfb9149e5f598b` +[kubernetes-client-darwin-386.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-client-darwin-386.tar.gz) | `d77d33c6d6357b99089f65e1c9ec3cabdcf526ec56e87bdee6b09a8c1b1f1b8f6f0ed6d32f2d3b352391da848afc945e5bb6bfc4c05d90fb4ba429e2d2c3ec0f` +[kubernetes-client-darwin-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-client-darwin-amd64.tar.gz) | `5b4a586defa2ba0ea7c8893dedfe48cae52a2cd324bcb311a3877e27493abb6cb76550e8201a9cac488cde9f83e0d30e6569b95641e8098fd9ec5df9c9e027b2` +[kubernetes-client-linux-386.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-client-linux-386.tar.gz) | `d50572fbb716393004ad2984a15043d2dfadedd16ae03a73fc85653266ae389071fd2c993923fbe9ea7fbd6b8cbeb6680ef147245e20f334969184d4b571509b` +[kubernetes-client-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-client-linux-amd64.tar.gz) | `12ab709e574228f170a2ee2686e18dcbfcf59f64599b2ab9047c2ed63f4bd23d6c9fc48104431c9fa616e0ba30041e1c44fb3994ab54c5c98c0c4a94c5ea4b80` +[kubernetes-client-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-client-linux-arm.tar.gz) | `3a8c75b62cf9e6476417246d4aaeda5a13b74bc073444fc3649198b9d5dc1e7a62aa6b914c7da5a42bcd6164a8f63aa8b256b3579979b6465afe5aa5533ce501` +[kubernetes-client-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-client-linux-arm64.tar.gz) | `0f5b5956850f11a826d59d226b6a22645ca1f63893cd33c17dfe004bd316f2704d800beb0b9c91a204efc125241825243c9e89b86c01e523bd07636ef925772e` +[kubernetes-client-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-client-linux-ppc64le.tar.gz) | `06c60dd2e4e8d1ab45474a5b85345b4f644d0c1c66e167596c6c91bd607f957b68121fdb7efed362cb6799e7bcf14752b01f8bea0c929deb85311180f11469be` +[kubernetes-client-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-client-linux-s390x.tar.gz) | `4630e9e523beb02d8d3900c71b3306561c2d119d588399c93d578184eb1a53601ceefe15a600740c13d565eaf24a679f17e371e2b19a70f77bbceab84acb58b3` +[kubernetes-client-windows-386.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-client-windows-386.tar.gz) | `0c0fcc9c492aceb00ff7fd3c10ba228c7bb10d6139b75ceecd8f85532797c5dc1162b39d94ebae5fa6b4c26f2b2a81630371617426f7537d9d11456943c7d50c` +[kubernetes-client-windows-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-client-windows-amd64.tar.gz) | `3548a6d8618c6c7c8042ae8c3eb69654314392c46f839de24ab72d9faa79993a6cf989f6ac619e418e817300081742c9928c8c2dd82cffc74f7c0e532e42288d` ### Server Binaries -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-server-linux-amd64.tar.gz) | `d9c282cd02c8c3fdbeb2f46abd0ddd257a8449e94be3beed2514c6e30a335a87` -[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-server-linux-arm.tar.gz) | `613390ba73f4236feb10bb4f70cbf96e504cf8d598da0180efc887d316b8bc5e` -[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-server-linux-arm64.tar.gz) | `1dd417f59d17c3583c6b4a3989d24c57e4989eb7b6ab9f2aa10c4cbf9bf5c11b` -[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-server-linux-ppc64le.tar.gz) | `44e9e6424ed3a5a91f5adefa456b2b71c0c5d3b01be9f60f5c8c0f958815ffc1` -[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-server-linux-s390x.tar.gz) | `3118d9c955f9a50f86ebba324894f06dbf7c1cb8f9bc5bdf6a95caf2a6678805` +[kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-server-linux-amd64.tar.gz) | `9dbf2343ef9539b7d4d73949bcd9eef6f46ece59e97fa3390a0e695d0cb2eacbbbae17e3ed53432a8018f55e6db2421a58739aacbc163776d8b2fe774ad62c34` +[kubernetes-server-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-server-linux-arm.tar.gz) | `a985f3c302246df9bff4b927a2596d209c19fb2f245aa5cb5de189b6a9d247d6fd0234edd45968a691f1a2714a0b72ffba2df1aebf361ba1a3461ab7e5fda2ff` +[kubernetes-server-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-server-linux-arm64.tar.gz) | `80d20df07e6a29b7aedccbd4e26c1c0565b2a1c3146e1a5bb2ebd2e8cf9ab063db137389a498fd6a6c3c42da43486186af6f65fba399b332e4cae134badd7ab0` +[kubernetes-server-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-server-linux-ppc64le.tar.gz) | `7d45ed3aa8b36e9e666b334ff3ed3de238caea34b4a92b5e1a61a6e7223ae8581bafff43a5b72447a43e118ad4b2c5c15aa0d6faab9a6a72a8fcf99abf340a1e` +[kubernetes-server-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-server-linux-s390x.tar.gz) | `30698478fab2fe7daccac97917b0b21b018c194ec39b005728f8cddf77f889aa3e1a520e0d1d681f9d8b7889a887aa6eb98c33cb04a8bf52b9a40bb8589aa34f` ### Node Binaries -filename | sha256 hash +filename | sha512 hash -------- | ----------- -[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-node-linux-amd64.tar.gz) | `6b4d363d190e0ce6f4e41d19a0ac350b39cad7859bc442166a1da9124d1a82bb` -[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-node-linux-arm.tar.gz) | `c80ac005c228217b871bf3e9de032044659db3aa048cc95b101820e31d62264c` -[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-node-linux-arm64.tar.gz) | `d8b84e7cc6ff5d0e26b045de37bdd40ca8809c303b601d8604902e5957d98621` -[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-node-linux-ppc64le.tar.gz) | `b0a667c5c905e6e724fba95d44797fb52afb564aedd1c25cbd4e632e152843e9` -[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-node-linux-s390x.tar.gz) | `78e7dbb82543ea6ac70767ed63c92823726adb6257f6b70b5911843d18288df7` -[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.12.0-alpha.1/kubernetes-node-windows-amd64.tar.gz) | `1a3e11cc3f1a0297de2b894a43eb56ede5fbd5cdc43e4da7e61171f5c1f3ef60` +[kubernetes-node-linux-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-node-linux-amd64.tar.gz) | `4497d14ac81677b43f0b75a457890c1f3bb8745a39875f58d53c734bec1947c37388228e0c952ca87f22d74af101a9263546db07bf1a021d59cba3cea1d5e5b9` +[kubernetes-node-linux-arm.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-node-linux-arm.tar.gz) | `a3b0357db50e0dec7b0474816fec287388adabc76cc309a40dee9bc73771c951e4526a06145a8b332d72e5999dabb5e467d00d7c47035559a79e56f863def2f7` +[kubernetes-node-linux-arm64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-node-linux-arm64.tar.gz) | `43af8ec4c5f2a1e2baa8cd13817e127fb6a3576dd811a30c4cc5f04d8a9a8bb2267eb5c42e0a895cf2ec0e3260b73c818249296c957625f4048e13102606680a` +[kubernetes-node-linux-ppc64le.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-node-linux-ppc64le.tar.gz) | `840354219b3e59ed05b5b44cbbf4d45ccc4c0d74044e28c8a557ca75d12e509b091eb10d9bed81e300cb484b0a0f735424383c8b266fbb3e2aa7a2d50dceaf9e` +[kubernetes-node-linux-s390x.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-node-linux-s390x.tar.gz) | `796ca2e6855bd942a9a63d93f847ae62c5ee74195e041b60b89ee7d0e5a75643a8809bfaa36898daa176bccf140a8e5e858f5cb74e457d8bbb0a650600628ceb` +[kubernetes-node-windows-amd64.tar.gz](https://dl.k8s.io/v1.13.0-alpha.1/kubernetes-node-windows-amd64.tar.gz) | `96d666e8446d09088bdcb440559035118dce07a2d9f5718856192fd807b61840d2d6cee2a808eb13f2e947784300edc1ac41ade14dc3536c044e4d08982f0fdd` -## Changelog since v1.11.0 +## Changelog since v1.12.0 ### Action Required -* action required: the API server and client-go libraries have been fixed to support additional non-alpha-numeric characters in UserInfo "extra" data keys. Both should be updated in order to properly support extra data containing "/" characters or other characters disallowed in HTTP headers. ([#65799](https://github.com/kubernetes/kubernetes/pull/65799), [@dekkagaijin](https://github.com/dekkagaijin)) -* [action required] The `NodeConfiguration` kind in the kubeadm v1alpha2 API has been renamed `JoinConfiguration` in v1alpha3 ([#65951](https://github.com/kubernetes/kubernetes/pull/65951), [@luxas](https://github.com/luxas)) -* ACTION REQUIRED: Removes defaulting of CSI file system type to ext4. All the production drivers listed under https://kubernetes-csi.github.io/docs/Drivers.html were inspected and should not be impacted after this change. If you are using a driver not in that list, please test the drivers on an updated test cluster first. ``` ([#65499](https://github.com/kubernetes/kubernetes/pull/65499), [@krunaljain](https://github.com/krunaljain)) -* [action required] The `MasterConfiguration` kind in the kubeadm v1alpha2 API has been renamed `InitConfiguration` in v1alpha3 ([#65945](https://github.com/kubernetes/kubernetes/pull/65945), [@luxas](https://github.com/luxas)) -* [action required] The formerly publicly-available cAdvisor web UI that the kubelet started using `--cadvisor-port` is now entirely removed in 1.12. The recommended way to run cAdvisor if you still need it, is via a DaemonSet. ([#65707](https://github.com/kubernetes/kubernetes/pull/65707), [@dims](https://github.com/dims)) -* Cluster Autoscaler version updated to 1.3.1-beta.1. Release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.3.1-beta.1 ([#65857](https://github.com/kubernetes/kubernetes/pull/65857), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) - * Default value for expendable pod priority cutoff in GCP deployment of Cluster Autoscaler changed from 0 to -10. - * action required: users deploying workloads with priority lower than 0 may want to use priority lower than -10 to avoid triggering scale-up. -* [action required] kubeadm: The `v1alpha1` config API has been removed. ([#65628](https://github.com/kubernetes/kubernetes/pull/65628), [@luxas](https://github.com/luxas)) - * Please convert your `v1alpha1` configuration files to `v1alpha2` using the - * `kubeadm config migrate` command of kubeadm v1.11.x -* kube-apiserver: the `Priority` admission plugin is now enabled by default when using `--enable-admission-plugins`. If using `--admission-control` to fully specify the set of admission plugins, the `Priority` admission plugin should be added if using the `PodPriority` feature, which is enabled by default in 1.11. ([#65739](https://github.com/kubernetes/kubernetes/pull/65739), [@liggitt](https://github.com/liggitt)) -* The `system-node-critical` and `system-cluster-critical` priority classes are now limited to the `kube-system` namespace by the `PodPriority` admission plugin. ([#65593](https://github.com/kubernetes/kubernetes/pull/65593), [@bsalamat](https://github.com/bsalamat)) -* kubernetes-worker juju charm: Added support for setting the --enable-ssl-chain-completion option on the ingress proxy. "action required": if your installation relies on supplying incomplete certificate chains and using OCSP to fill them in, you must set "ingress-ssl-chain-completion" to "true" in your juju configuration. ([#63845](https://github.com/kubernetes/kubernetes/pull/63845), [@paulgear](https://github.com/paulgear)) +* kube-apiserver: the deprecated `--etcd-quorum-read` flag has been removed, and quorum reads are always enabled when fetching data from etcd. ([#69527](https://github.com/kubernetes/kubernetes/pull/69527), [@liggitt](https://github.com/liggitt)) +* Moved staging/src/k8s.io/client-go/tools/bootstrap to staging/src/k8s… ([#67356](https://github.com/kubernetes/kubernetes/pull/67356), [@yliaog](https://github.com/yliaog)) +* [action required] kubeadm: The `v1alpha2` config API has been removed. ([#69055](https://github.com/kubernetes/kubernetes/pull/69055), [@fabriziopandini](https://github.com/fabriziopandini)) + * Please convert your `v1alpha2` configuration files to `v1alpha3` using the + * `kubeadm config migrate` command of kubeadm v1.12.x ### Other notable changes -* admin RBAC role now aggregates edit and view. edit RBAC role now aggregates view. ([#66684](https://github.com/kubernetes/kubernetes/pull/66684), [@deads2k](https://github.com/deads2k)) -* Speed up HPA reaction to metric changes by removing scale up forbidden window. ([#66615](https://github.com/kubernetes/kubernetes/pull/66615), [@jbartosik](https://github.com/jbartosik)) - * Scale up forbidden window was protecting HPA against making decision to scale up based on metrics gathered during pod initialisation (which may be invalid, for example pod may be using a lot of CPU despite not doing any "actual" work). - * To avoid that negative effect only use per pod metrics from pods that are: - * - ready (so metrics about them should be valid), or - * - unready but creation and last readiness change timestamps are apart more than 10s (pods that have formerly been ready and so metrics are in at least some cases (pod becoming unready because of overload) very useful). -* The `kubectl patch` command no longer exits with exit code 1 when a redundant patch results in a no-op ([#66725](https://github.com/kubernetes/kubernetes/pull/66725), [@juanvallejo](https://github.com/juanvallejo)) -* Improved the output of `kubectl get events` to prioritize showing the message, and move some fields to `-o wide`. ([#66643](https://github.com/kubernetes/kubernetes/pull/66643), [@smarterclayton](https://github.com/smarterclayton)) -* Added CPU Manager state validation in case of changed CPU topology. ([#66718](https://github.com/kubernetes/kubernetes/pull/66718), [@ipuustin](https://github.com/ipuustin)) -* Make EBS volume expansion faster ([#66728](https://github.com/kubernetes/kubernetes/pull/66728), [@gnufied](https://github.com/gnufied)) -* Kubelet serving certificate bootstrapping and rotation has been promoted to beta status. ([#66726](https://github.com/kubernetes/kubernetes/pull/66726), [@liggitt](https://github.com/liggitt)) -* Flag --pod (-p shorthand) of kubectl exec command marked as deprecated ([#66558](https://github.com/kubernetes/kubernetes/pull/66558), [@quasoft](https://github.com/quasoft)) -* Fixed an issue which prevented `gcloud` from working on GCE when metadata concealment was enabled. ([#66630](https://github.com/kubernetes/kubernetes/pull/66630), [@dekkagaijin](https://github.com/dekkagaijin)) -* Azure Go SDK has been upgraded to v19.0.0 and VirtualMachineScaleSetVM now supports availability zones. ([#66648](https://github.com/kubernetes/kubernetes/pull/66648), [@feiskyer](https://github.com/feiskyer)) -* kubeadm now can join the cluster with pre-existing client certificate if provided ([#66482](https://github.com/kubernetes/kubernetes/pull/66482), [@dixudx](https://github.com/dixudx)) -* If `TaintNodesByCondition` enabled, taint node with `TaintNodeUnschedulable` when ([#63955](https://github.com/kubernetes/kubernetes/pull/63955), [@k82cn](https://github.com/k82cn)) - * initializing node to avoid race condition. -* kubeadm: remove misleading error message regarding image pulling ([#66658](https://github.com/kubernetes/kubernetes/pull/66658), [@dixudx](https://github.com/dixudx)) -* Fix Stackdriver integration based on node annotation container.googleapis.com/instance_id. ([#66676](https://github.com/kubernetes/kubernetes/pull/66676), [@kawych](https://github.com/kawych)) -* Fix kubelet startup failure when using ExecPlugin in kubeconfig ([#66395](https://github.com/kubernetes/kubernetes/pull/66395), [@awly](https://github.com/awly)) -* When attaching iSCSI volumes, kubelet now scans only the specific ([#63176](https://github.com/kubernetes/kubernetes/pull/63176), [@bswartz](https://github.com/bswartz)) - * LUNs being attached, and also deletes them after detaching. This avoids - * dangling references to LUNs that no longer exist, which used to be the - * cause of random I/O errors/timeouts in kernel logs, slowdowns during - * block-device related operations, and very rare cases of data corruption. -* kubeadm: Pull sidecar and dnsmasq-nanny images when using kube-dns ([#66499](https://github.com/kubernetes/kubernetes/pull/66499), [@rosti](https://github.com/rosti)) -* Extender preemption should respect IsInterested() ([#66291](https://github.com/kubernetes/kubernetes/pull/66291), [@resouer](https://github.com/resouer)) -* Properly autopopulate OpenAPI version field without needing other OpenAPI fields present in generic API server code. ([#66411](https://github.com/kubernetes/kubernetes/pull/66411), [@DirectXMan12](https://github.com/DirectXMan12)) -* renamed command line option --cri-socket-path of the kubeadm subcommand "kubeadm config images pull" to --cri-socket to be consistent with the rest of kubeadm subcommands. ([#66382](https://github.com/kubernetes/kubernetes/pull/66382), [@bart0sh](https://github.com/bart0sh)) -* The --docker-disable-shared-pid kubelet flag has been removed. PID namespace sharing can instead be enable per-pod using the ShareProcessNamespace option. ([#66506](https://github.com/kubernetes/kubernetes/pull/66506), [@verb](https://github.com/verb)) -* Add support for using User Assigned MSI (https://docs.microsoft.com/en-us/azure/active-directory/managed-service-identity/overview) with Kubernetes cluster on Azure. ([#66180](https://github.com/kubernetes/kubernetes/pull/66180), [@kkmsft](https://github.com/kkmsft)) -* fix acr could not be listed in sp issue ([#66429](https://github.com/kubernetes/kubernetes/pull/66429), [@andyzhangx](https://github.com/andyzhangx)) -* This PR will leverage subtests on the existing table tests for the scheduler units. ([#63665](https://github.com/kubernetes/kubernetes/pull/63665), [@xchapter7x](https://github.com/xchapter7x)) - * Some refactoring of error/status messages and functions to align with new approach. -* Fix volume limit for EBS on m5 and c5 instance types ([#66397](https://github.com/kubernetes/kubernetes/pull/66397), [@gnufied](https://github.com/gnufied)) -* Extend TLS timeouts to work around slow arm64 math/big ([#66264](https://github.com/kubernetes/kubernetes/pull/66264), [@joejulian](https://github.com/joejulian)) -* kubeadm: stop setting UID in the kubelet ConfigMap ([#66341](https://github.com/kubernetes/kubernetes/pull/66341), [@runiq](https://github.com/runiq)) -* kubectl: fixes a panic displaying pods with nominatedNodeName set ([#66406](https://github.com/kubernetes/kubernetes/pull/66406), [@liggitt](https://github.com/liggitt)) -* Update crictl to v1.11.1. ([#66152](https://github.com/kubernetes/kubernetes/pull/66152), [@Random-Liu](https://github.com/Random-Liu)) -* fixes a panic when using a mutating webhook admission plugin with a DELETE operation ([#66425](https://github.com/kubernetes/kubernetes/pull/66425), [@liggitt](https://github.com/liggitt)) -* GCE: Fixes loadbalancer creation and deletion issues appearing in 1.10.5. ([#66400](https://github.com/kubernetes/kubernetes/pull/66400), [@nicksardo](https://github.com/nicksardo)) -* Azure nodes with availability zone now will have label `failure-domain.beta.kubernetes.io/zone=-`. ([#66242](https://github.com/kubernetes/kubernetes/pull/66242), [@feiskyer](https://github.com/feiskyer)) -* Re-design equivalence class cache to two level cache ([#65714](https://github.com/kubernetes/kubernetes/pull/65714), [@resouer](https://github.com/resouer)) -* Checks CREATE admission for create-on-update requests instead of UPDATE admission ([#65572](https://github.com/kubernetes/kubernetes/pull/65572), [@yue9944882](https://github.com/yue9944882)) -* This PR will leverage subtests on the existing table tests for the scheduler units. ([#63666](https://github.com/kubernetes/kubernetes/pull/63666), [@xchapter7x](https://github.com/xchapter7x)) - * Some refactoring of error/status messages and functions to align with new approach. -* Fixed a panic in the node status update logic when existing node has nil labels. ([#66307](https://github.com/kubernetes/kubernetes/pull/66307), [@guoshimin](https://github.com/guoshimin)) -* Bump Ingress-gce version to 1.2.0 ([#65641](https://github.com/kubernetes/kubernetes/pull/65641), [@freehan](https://github.com/freehan)) -* Bump event-exporter to 0.2.2 to pick up security fixes. ([#66157](https://github.com/kubernetes/kubernetes/pull/66157), [@loburm](https://github.com/loburm)) -* Allow ScaleIO volumes to be provisioned without having to first manually create /dev/disk/by-id path on each kubernetes node (if not already present) ([#66174](https://github.com/kubernetes/kubernetes/pull/66174), [@ddebroy](https://github.com/ddebroy)) -* fix rollout status for statefulsets ([#62943](https://github.com/kubernetes/kubernetes/pull/62943), [@faraazkhan](https://github.com/faraazkhan)) -* Fix for resourcepool-path configuration in the vsphere.conf file. ([#66261](https://github.com/kubernetes/kubernetes/pull/66261), [@divyenpatel](https://github.com/divyenpatel)) -* OpenAPI spec and documentation reflect 202 Accepted response path for delete request ([#63418](https://github.com/kubernetes/kubernetes/pull/63418), [@roycaihw](https://github.com/roycaihw)) -* fixes a validation error that could prevent updates to StatefulSet objects containing non-normalized resource requests ([#66165](https://github.com/kubernetes/kubernetes/pull/66165), [@liggitt](https://github.com/liggitt)) -* Fix validation for HealthzBindAddress in kube-proxy when --healthz-port is set to 0 ([#66138](https://github.com/kubernetes/kubernetes/pull/66138), [@wsong](https://github.com/wsong)) -* kubeadm: use an HTTP request timeout when fetching the latest version of Kubernetes from dl.k8s.io ([#65676](https://github.com/kubernetes/kubernetes/pull/65676), [@dkoshkin](https://github.com/dkoshkin)) -* Support configuring the Azure load balancer idle connection timeout for services ([#66045](https://github.com/kubernetes/kubernetes/pull/66045), [@cpuguy83](https://github.com/cpuguy83)) -* `kubectl config set-context` can now set attributes of the current context, like the current namespace, by passing `--current` instead of a specific context name ([#66140](https://github.com/kubernetes/kubernetes/pull/66140), [@liggitt](https://github.com/liggitt)) -* The alpha `Initializers` admission plugin is no longer enabled by default. This matches the off-by-default behavior of the alpha API which drives initializer behavior. ([#66039](https://github.com/kubernetes/kubernetes/pull/66039), [@liggitt](https://github.com/liggitt)) -* kubeadm: Default component configs are printable via kubeadm config print-default ([#66074](https://github.com/kubernetes/kubernetes/pull/66074), [@rosti](https://github.com/rosti)) -* prevents infinite CLI wait on delete when item is recreated ([#66136](https://github.com/kubernetes/kubernetes/pull/66136), [@deads2k](https://github.com/deads2k)) -* Preserve vmUUID when renewing nodeinfo in vSphere cloud provider ([#66007](https://github.com/kubernetes/kubernetes/pull/66007), [@w-leads](https://github.com/w-leads)) -* Cluster Autoscaler version updated to 1.3.1. Release notes: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.3.1 ([#66122](https://github.com/kubernetes/kubernetes/pull/66122), [@aleksandra-malinowska](https://github.com/aleksandra-malinowska)) -* Expose docker registry config for addons used in Juju deployments ([#66092](https://github.com/kubernetes/kubernetes/pull/66092), [@kwmonroe](https://github.com/kwmonroe)) -* kubelets that specify `--cloud-provider` now only report addresses in Node status as determined by the cloud provider ([#65594](https://github.com/kubernetes/kubernetes/pull/65594), [@liggitt](https://github.com/liggitt)) - * kubelet serving certificate rotation now reacts to changes in reported node addresses, and will request certificates for addresses set by an external cloud provider -* Fix the bug where image garbage collection is disabled by mistake. ([#66051](https://github.com/kubernetes/kubernetes/pull/66051), [@jiaxuanzhou](https://github.com/jiaxuanzhou)) -* fixes an issue with multi-line annotations injected via downward API files getting scrambled ([#65992](https://github.com/kubernetes/kubernetes/pull/65992), [@liggitt](https://github.com/liggitt)) -* kubeadm: run kube-proxy on non-master tainted nodes ([#65931](https://github.com/kubernetes/kubernetes/pull/65931), [@neolit123](https://github.com/neolit123)) -* "kubectl delete" no longer waits for dependent objects to be deleted when removing parent resources ([#65908](https://github.com/kubernetes/kubernetes/pull/65908), [@juanvallejo](https://github.com/juanvallejo)) -* Introduce a new flag `--keepalive` for kubectl proxy to allow setting keep-alive period for long-running request. ([#63793](https://github.com/kubernetes/kubernetes/pull/63793), [@hzxuzhonghu](https://github.com/hzxuzhonghu)) -* If Openstack LoadBalancer is not defined in cloud config, the loadbalancer is not initialized any more in openstack. All setups must have some setting under that section ([#65781](https://github.com/kubernetes/kubernetes/pull/65781), [@zetaab](https://github.com/zetaab)) -* Re-adds `pkg/generated/bindata.go` to the repository to allow some parts of k8s.io/kubernetes to be go-vendorable. ([#65985](https://github.com/kubernetes/kubernetes/pull/65985), [@ixdy](https://github.com/ixdy)) -* Fix a bug that preempting a pod may block forever. ([#65987](https://github.com/kubernetes/kubernetes/pull/65987), [@Random-Liu](https://github.com/Random-Liu)) -* Fix flexvolume in containarized kubelets ([#65549](https://github.com/kubernetes/kubernetes/pull/65549), [@gnufied](https://github.com/gnufied)) -* Add volume mode filed to constructed volume spec for CSI plugin ([#65456](https://github.com/kubernetes/kubernetes/pull/65456), [@wenlxie](https://github.com/wenlxie)) -* Fix an issue with dropped audit logs, when truncating and batch backends enabled at the same time. ([#65823](https://github.com/kubernetes/kubernetes/pull/65823), [@loburm](https://github.com/loburm)) -* Support traffic shaping for CNI network driver ([#63194](https://github.com/kubernetes/kubernetes/pull/63194), [@m1093782566](https://github.com/m1093782566)) -* kubeadm: Use separate YAML documents for the kubelet and kube-proxy ComponentConfigs ([#65787](https://github.com/kubernetes/kubernetes/pull/65787), [@luxas](https://github.com/luxas)) -* kubeadm: Fix pause image to not use architecture, as it is a manifest list ([#65920](https://github.com/kubernetes/kubernetes/pull/65920), [@dims](https://github.com/dims)) -* kubeadm: print required flags when running kubeadm upgrade plan ([#65802](https://github.com/kubernetes/kubernetes/pull/65802), [@xlgao-zju](https://github.com/xlgao-zju)) -* Fix `RunAsGroup` which doesn't work since 1.10. ([#65926](https://github.com/kubernetes/kubernetes/pull/65926), [@Random-Liu](https://github.com/Random-Liu)) -* Running `kubectl describe pvc` now shows which pods are mounted to the pvc being described with the `Mounted By` field ([#65837](https://github.com/kubernetes/kubernetes/pull/65837), [@clandry94](https://github.com/clandry94)) -* fix azure storage account creation failure ([#65846](https://github.com/kubernetes/kubernetes/pull/65846), [@andyzhangx](https://github.com/andyzhangx)) -* Allow kube- and cloud-controller-manager to listen on ports up to 65535. ([#65860](https://github.com/kubernetes/kubernetes/pull/65860), [@sttts](https://github.com/sttts)) -* Allow kube-scheduler to listen on ports up to 65535. ([#65833](https://github.com/kubernetes/kubernetes/pull/65833), [@sttts](https://github.com/sttts)) -* kubeadm: Remove usage of `PersistentVolumeLabel` ([#65827](https://github.com/kubernetes/kubernetes/pull/65827), [@xlgao-zju](https://github.com/xlgao-zju)) -* kubeadm: Add a `v1alpha3` API. ([#65629](https://github.com/kubernetes/kubernetes/pull/65629), [@luxas](https://github.com/luxas)) -* Update to use go1.10.3 ([#65726](https://github.com/kubernetes/kubernetes/pull/65726), [@ixdy](https://github.com/ixdy)) -* LimitRange and Endpoints resources can be created via an update API call if the object does not already exist. When this occurs, an authorization check is now made to ensure the user making the API call is authorized to create the object. In previous releases, only an update authorization check was performed. ([#65150](https://github.com/kubernetes/kubernetes/pull/65150), [@jennybuckley](https://github.com/jennybuckley)) -* Fix 'kubectl cp' with no arguments causes a panic ([#65482](https://github.com/kubernetes/kubernetes/pull/65482), [@wgliang](https://github.com/wgliang)) -* bazel deb package bugfix: The kubeadm deb package now reloads the kubelet after installation ([#65554](https://github.com/kubernetes/kubernetes/pull/65554), [@rdodev](https://github.com/rdodev)) -* fix smb mount issue ([#65751](https://github.com/kubernetes/kubernetes/pull/65751), [@andyzhangx](https://github.com/andyzhangx)) -* More fields are allowed at the root of the CRD validation schema when the status subresource is enabled. ([#65357](https://github.com/kubernetes/kubernetes/pull/65357), [@nikhita](https://github.com/nikhita)) -* Reload systemd config files before starting kubelet. ([#65702](https://github.com/kubernetes/kubernetes/pull/65702), [@mborsz](https://github.com/mborsz)) -* Unix: support ZFS as a valid graph driver for Docker ([#65635](https://github.com/kubernetes/kubernetes/pull/65635), [@neolit123](https://github.com/neolit123)) -* Fix controller-manager crashes when flex plugin is removed from flex plugin directory ([#65536](https://github.com/kubernetes/kubernetes/pull/65536), [@gnufied](https://github.com/gnufied)) -* Enable etcdv3 client prometheus metics ([#64741](https://github.com/kubernetes/kubernetes/pull/64741), [@wgliang](https://github.com/wgliang)) -* skip nodes that have a primary NIC in a 'Failed' provisioningState ([#65412](https://github.com/kubernetes/kubernetes/pull/65412), [@yastij](https://github.com/yastij)) -* kubeadm: remove redundant flags settings for kubelet ([#64682](https://github.com/kubernetes/kubernetes/pull/64682), [@dixudx](https://github.com/dixudx)) -* Fixes the wrong elasticsearch node counter ([#65627](https://github.com/kubernetes/kubernetes/pull/65627), [@IvanovOleg](https://github.com/IvanovOleg)) -* - Can configure the vsphere cloud provider with a trusted Root-CA ([#64758](https://github.com/kubernetes/kubernetes/pull/64758), [@mariantalla](https://github.com/mariantalla)) -* Add Ubuntu 18.04 (Bionic) series to Juju charms ([#65644](https://github.com/kubernetes/kubernetes/pull/65644), [@tvansteenburgh](https://github.com/tvansteenburgh)) -* Fix local volume directory can't be deleted because of volumeMode error ([#65310](https://github.com/kubernetes/kubernetes/pull/65310), [@wenlxie](https://github.com/wenlxie)) -* kubectl: --use-openapi-print-columns is deprecated in favor of --server-print ([#65601](https://github.com/kubernetes/kubernetes/pull/65601), [@liggitt](https://github.com/liggitt)) -* Add prometheus scrape port to CoreDNS service ([#65589](https://github.com/kubernetes/kubernetes/pull/65589), [@rajansandeep](https://github.com/rajansandeep)) -* fixes an out of range panic in the NoExecuteTaintManager controller when running a non-64-bit build ([#65596](https://github.com/kubernetes/kubernetes/pull/65596), [@liggitt](https://github.com/liggitt)) -* kubectl: fixes a regression with --use-openapi-print-columns that would not print object contents ([#65600](https://github.com/kubernetes/kubernetes/pull/65600), [@liggitt](https://github.com/liggitt)) -* Hostnames are now converted to lowercase before being used for node lookups in the kubernetes-worker charm. ([#65487](https://github.com/kubernetes/kubernetes/pull/65487), [@dshcherb](https://github.com/dshcherb)) -* N/A ([#64660](https://github.com/kubernetes/kubernetes/pull/64660), [@figo](https://github.com/figo)) -* bugfix: Do not print feature gates in the generic apiserver code for glog level 0 ([#65584](https://github.com/kubernetes/kubernetes/pull/65584), [@neolit123](https://github.com/neolit123)) -* Add metrics for PVC in-use ([#64527](https://github.com/kubernetes/kubernetes/pull/64527), [@gnufied](https://github.com/gnufied)) -* Fixed exception detection in fluentd-gcp plugin. ([#65361](https://github.com/kubernetes/kubernetes/pull/65361), [@xperimental](https://github.com/xperimental)) -* api-machinery utility functions `SetTransportDefaults` and `DialerFor` once again respect custom Dial functions set on transports ([#65547](https://github.com/kubernetes/kubernetes/pull/65547), [@liggitt](https://github.com/liggitt)) -* Improve the display of jobs in `kubectl get` and `kubectl describe` to emphasize progress and duration. ([#65463](https://github.com/kubernetes/kubernetes/pull/65463), [@smarterclayton](https://github.com/smarterclayton)) -* kubectl convert previous created a list inside of a list. Now it is only wrapped once. ([#65489](https://github.com/kubernetes/kubernetes/pull/65489), [@deads2k](https://github.com/deads2k)) -* fix azure disk creation issue when specifying external resource group ([#65516](https://github.com/kubernetes/kubernetes/pull/65516), [@andyzhangx](https://github.com/andyzhangx)) -* fixes a regression in kube-scheduler to properly load client connection information from a `--config` file that references a kubeconfig file ([#65507](https://github.com/kubernetes/kubernetes/pull/65507), [@liggitt](https://github.com/liggitt)) -* Fixed cleanup of CSI metadata files. ([#65323](https://github.com/kubernetes/kubernetes/pull/65323), [@jsafrane](https://github.com/jsafrane)) -* Update Rescheduler's manifest to use version 0.4.0. ([#65454](https://github.com/kubernetes/kubernetes/pull/65454), [@bsalamat](https://github.com/bsalamat)) -* On COS, NPD creates a node condition for frequent occurrences of unregister_netdevice ([#65342](https://github.com/kubernetes/kubernetes/pull/65342), [@dashpole](https://github.com/dashpole)) -* Properly manage security groups for loadbalancer services on OpenStack. ([#65373](https://github.com/kubernetes/kubernetes/pull/65373), [@multi-io](https://github.com/multi-io)) -* Add user-agent to audit-logging. ([#64812](https://github.com/kubernetes/kubernetes/pull/64812), [@hzxuzhonghu](https://github.com/hzxuzhonghu)) -* kubeadm: notify the user of manifest upgrade timeouts ([#65164](https://github.com/kubernetes/kubernetes/pull/65164), [@xlgao-zju](https://github.com/xlgao-zju)) -* Fixes incompatibility with custom scheduler extender configurations specifying `bindVerb` ([#65424](https://github.com/kubernetes/kubernetes/pull/65424), [@liggitt](https://github.com/liggitt)) -* Using `kubectl describe` on CRDs that use underscores will be prettier. ([#65391](https://github.com/kubernetes/kubernetes/pull/65391), [@smarterclayton](https://github.com/smarterclayton)) -* Improve scheduler's performance by eliminating sorting of nodes by their score. ([#65396](https://github.com/kubernetes/kubernetes/pull/65396), [@bsalamat](https://github.com/bsalamat)) -* Add more conditions to the list of predicate failures that won't be resolved by preemption. ([#64995](https://github.com/kubernetes/kubernetes/pull/64995), [@bsalamat](https://github.com/bsalamat)) -* Allow access to ClusterIP from the host network namespace when kube-proxy is started in IPVS mode without either masqueradeAll or clusterCIDR flags ([#65388](https://github.com/kubernetes/kubernetes/pull/65388), [@lbernail](https://github.com/lbernail)) -* User can now use `sudo crictl` on GCE cluster. ([#65389](https://github.com/kubernetes/kubernetes/pull/65389), [@Random-Liu](https://github.com/Random-Liu)) -* Tolerate missing watch permission when deleting a resource ([#65370](https://github.com/kubernetes/kubernetes/pull/65370), [@deads2k](https://github.com/deads2k)) -* Prevents a `kubectl delete` hang when deleting controller managed lists ([#65367](https://github.com/kubernetes/kubernetes/pull/65367), [@deads2k](https://github.com/deads2k)) -* fixes a memory leak in the kube-controller-manager observed when large numbers of pods with tolerations are created/deleted ([#65339](https://github.com/kubernetes/kubernetes/pull/65339), [@liggitt](https://github.com/liggitt)) -* checkLimitsForResolvConf for the pod create and update events instead of checking period ([#64860](https://github.com/kubernetes/kubernetes/pull/64860), [@wgliang](https://github.com/wgliang)) -* Fix concurrent map access panic ([#65334](https://github.com/kubernetes/kubernetes/pull/65334), [@dashpole](https://github.com/dashpole)) - * Don't watch .mount cgroups to reduce number of inotify watches - * Fix NVML initialization race condition - * Fix brtfs disk metrics when using a subdirectory of a subvolume -* Change Azure ARM Rate limiting error message. ([#65292](https://github.com/kubernetes/kubernetes/pull/65292), [@wgliang](https://github.com/wgliang)) -* AWS now checks for validity of ecryption key when creating encrypted volumes. Dynamic provisioning of encrypted volume may get slower due to these checks. ([#65223](https://github.com/kubernetes/kubernetes/pull/65223), [@jsafrane](https://github.com/jsafrane)) -* Report accurate status for kubernetes-master and -worker charms. ([#65187](https://github.com/kubernetes/kubernetes/pull/65187), [@kwmonroe](https://github.com/kwmonroe)) -* Fixed issue 63608, which is that under rare circumstances the ResourceQuota admission controller could lose track of an request in progress and time out after waiting 10 seconds for a decision to be made. ([#64598](https://github.com/kubernetes/kubernetes/pull/64598), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) -* In the vSphere cloud provider the `Global.vm-uuid` configuration option is not deprecated anymore, it can be used to overwrite the VMUUID on the controller-manager ([#65152](https://github.com/kubernetes/kubernetes/pull/65152), [@alvaroaleman](https://github.com/alvaroaleman)) -* fluentd-gcp grace termination period increased to 60s. ([#65084](https://github.com/kubernetes/kubernetes/pull/65084), [@x13n](https://github.com/x13n)) -* Pass cluster_location argument to Heapster ([#65176](https://github.com/kubernetes/kubernetes/pull/65176), [@kawych](https://github.com/kawych)) -* Fix a scalability issue where high rates of event writes degraded etcd performance. ([#64539](https://github.com/kubernetes/kubernetes/pull/64539), [@ccding](https://github.com/ccding)) -* Corrected a mistake in the documentation for wait.PollImmediate(...) ([#65026](https://github.com/kubernetes/kubernetes/pull/65026), [@spew](https://github.com/spew)) -* Split 'scheduling_latency_seconds' metric into finer steps (predicate, priority, premption) ([#65306](https://github.com/kubernetes/kubernetes/pull/65306), [@shyamjvs](https://github.com/shyamjvs)) -* Etcd health checks by the apiserver now ensure the apiserver can connect to and exercise the etcd API ([#65027](https://github.com/kubernetes/kubernetes/pull/65027), [@liggitt](https://github.com/liggitt)) -* Add e2e regression tests for the kubelet being secure ([#64140](https://github.com/kubernetes/kubernetes/pull/64140), [@dixudx](https://github.com/dixudx)) -* set EnableHTTPSTrafficOnly in azure storage account creation ([#64957](https://github.com/kubernetes/kubernetes/pull/64957), [@andyzhangx](https://github.com/andyzhangx)) -* Fixes an issue where Portworx PVCs remain in pending state when created using a StorageClass with empty parameters ([#64895](https://github.com/kubernetes/kubernetes/pull/64895), [@harsh-px](https://github.com/harsh-px)) -* This PR will leverage subtests on the existing table tests for the scheduler units. ([#63662](https://github.com/kubernetes/kubernetes/pull/63662), [@xchapter7x](https://github.com/xchapter7x)) - * Some refactoring of error/status messages and functions to align with new approach. -* This PR will leverage subtests on the existing table tests for the scheduler units. ([#63661](https://github.com/kubernetes/kubernetes/pull/63661), [@xchapter7x](https://github.com/xchapter7x)) - * Some refactoring of error/status messages and functions to align with new approach. -* This PR will leverage subtests on the existing table tests for the scheduler units. ([#63660](https://github.com/kubernetes/kubernetes/pull/63660), [@xchapter7x](https://github.com/xchapter7x)) - * Some refactoring of error/status messages and functions to align with new approach. -* Updated default image for nginx ingress in CDK to match current Kubernetes docs. ([#64285](https://github.com/kubernetes/kubernetes/pull/64285), [@hyperbolic2346](https://github.com/hyperbolic2346)) -* Added block volume support to Cinder volume plugin. ([#64879](https://github.com/kubernetes/kubernetes/pull/64879), [@bertinatto](https://github.com/bertinatto)) -* fixed incorrect OpenAPI schema for CustomResourceDefinition objects ([#65256](https://github.com/kubernetes/kubernetes/pull/65256), [@liggitt](https://github.com/liggitt)) -* ignore not found file error when watching manifests ([#64880](https://github.com/kubernetes/kubernetes/pull/64880), [@dixudx](https://github.com/dixudx)) -* add port-forward examples for sevice ([#64773](https://github.com/kubernetes/kubernetes/pull/64773), [@MasayaAoyama](https://github.com/MasayaAoyama)) -* Fix issues for block device not mapped to container. ([#64555](https://github.com/kubernetes/kubernetes/pull/64555), [@wenlxie](https://github.com/wenlxie)) -* Update crictl on GCE to v1.11.0. ([#65254](https://github.com/kubernetes/kubernetes/pull/65254), [@Random-Liu](https://github.com/Random-Liu)) -* Fixes missing nodes lines when kubectl top nodes ([#64389](https://github.com/kubernetes/kubernetes/pull/64389), [@yue9944882](https://github.com/yue9944882)) -* keep pod state consistent when scheduler cache UpdatePod ([#64692](https://github.com/kubernetes/kubernetes/pull/64692), [@adohe](https://github.com/adohe)) -* add external resource group support for azure disk ([#64427](https://github.com/kubernetes/kubernetes/pull/64427), [@andyzhangx](https://github.com/andyzhangx)) -* Increase the gRPC max message size to 16MB in the remote container runtime. ([#64672](https://github.com/kubernetes/kubernetes/pull/64672), [@mcluseau](https://github.com/mcluseau)) -* The new default value for the --allow-privileged parameter of the Kubernetes-worker charm has been set to true based on changes which went into the Kubernetes 1.10 release. Before this change the default value was set to false. If you're installing Canonical Kubernetes you should expect this value to now be true by default and you should now look to use PSP (pod security policies). ([#64104](https://github.com/kubernetes/kubernetes/pull/64104), [@CalvinHartwell](https://github.com/CalvinHartwell)) -* The --remove-extra-subjects and --remove-extra-permissions flags have been enabled for kubectl auth reconcile ([#64541](https://github.com/kubernetes/kubernetes/pull/64541), [@mrogers950](https://github.com/mrogers950)) -* Fix kubectl drain --timeout option when eviction is used. ([#64378](https://github.com/kubernetes/kubernetes/pull/64378), [@wrdls](https://github.com/wrdls)) -* This PR will leverage subtests on the existing table tests for the scheduler units. ([#63659](https://github.com/kubernetes/kubernetes/pull/63659), [@xchapter7x](https://github.com/xchapter7x)) - * Some refactoring of error/status messages and functions to align with new approach. +* Refactor factory_test.go to use a fake k8s client. ([#69412](https://github.com/kubernetes/kubernetes/pull/69412), [@tossmilestone](https://github.com/tossmilestone)) +* kubeadm: fix a case where fetching a kubernetesVersion from the internet still happened even if some commands don't need it. ([#69645](https://github.com/kubernetes/kubernetes/pull/69645), [@neolit123](https://github.com/neolit123)) +* Add tolerations for Stackdriver Logging and Metadata Agents. ([#69737](https://github.com/kubernetes/kubernetes/pull/69737), [@qingling128](https://github.com/qingling128)) +* Fix a bug in the scheduler that could cause the scheduler to go to an infinite loop when all nodes in a zone are removed. ([#69758](https://github.com/kubernetes/kubernetes/pull/69758), [@bsalamat](https://github.com/bsalamat)) +* Dry-run is promoted to Beta and will be enabled by default. ([#69644](https://github.com/kubernetes/kubernetes/pull/69644), [@apelisse](https://github.com/apelisse)) +* `kubectl get priorityclass` now prints value column by default. ([#69431](https://github.com/kubernetes/kubernetes/pull/69431), [@Huang-Wei](https://github.com/Huang-Wei)) +* Added a new container based image for running e2e tests ([#69368](https://github.com/kubernetes/kubernetes/pull/69368), [@dims](https://github.com/dims)) +* Remove the deprecated --google-json-key flag from kubelet. ([#69354](https://github.com/kubernetes/kubernetes/pull/69354), [@yujuhong](https://github.com/yujuhong)) +* kube-apiserver: fixes `procMount` field incorrectly being marked as required in openapi schema ([#69694](https://github.com/kubernetes/kubernetes/pull/69694), [@jessfraz](https://github.com/jessfraz)) +* The `LC_ALL` and `LC_MESSAGES` env vars can now be used to set desired locale for `kubectl` while keeping `LANG` unchanged. ([#69500](https://github.com/kubernetes/kubernetes/pull/69500), [@m1kola](https://github.com/m1kola)) +* Add ability to control primary GID of containers through Pod Spec and PodSecurityPolicy ([#67802](https://github.com/kubernetes/kubernetes/pull/67802), [@krmayankk](https://github.com/krmayankk)) +* NodeLifecycleController: Now node lease renewal is treated as the heartbeat signal from the node, in addition to NodeStatus Update. ([#69241](https://github.com/kubernetes/kubernetes/pull/69241), [@wangzhen127](https://github.com/wangzhen127)) +* [GCE] Enable by default audit logging truncating backend. ([#68288](https://github.com/kubernetes/kubernetes/pull/68288), [@loburm](https://github.com/loburm)) +* Enable insertId generation, and update Stackdriver Logging Agent image to 0.5-1.5.36-1-k8s. This help reduce log duplication and guarantee log order. ([#68920](https://github.com/kubernetes/kubernetes/pull/68920), [@qingling128](https://github.com/qingling128)) +* Move NodeInfo utils into pkg/scheduler/cache. ([#69495](https://github.com/kubernetes/kubernetes/pull/69495), [@wgliang](https://github.com/wgliang)) +* adds dynamic shared informers to write generic, non-generated controllers ([#69308](https://github.com/kubernetes/kubernetes/pull/69308), [@p0lyn0mial](https://github.com/p0lyn0mial)) +* Move CacheComparer to pkg/scheduler/internal/cache/comparer. ([#69317](https://github.com/kubernetes/kubernetes/pull/69317), [@wgliang](https://github.com/wgliang)) +* Updating OWNERS list for vSphere Cloud Provider. ([#69187](https://github.com/kubernetes/kubernetes/pull/69187), [@SandeepPissay](https://github.com/SandeepPissay)) +* The default storage class annotation for the storage addons has been changed to use the GA variant ([#68345](https://github.com/kubernetes/kubernetes/pull/68345), [@smelchior](https://github.com/smelchior)) +* Upgrade to etcd 3.3 client ([#69322](https://github.com/kubernetes/kubernetes/pull/69322), [@jpbetz](https://github.com/jpbetz)) +* fix GetVolumeLimits log flushing issue ([#69558](https://github.com/kubernetes/kubernetes/pull/69558), [@andyzhangx](https://github.com/andyzhangx)) +* It is now possible to use named ports in the `kubectl port-forward` command ([#69477](https://github.com/kubernetes/kubernetes/pull/69477), [@m1kola](https://github.com/m1kola)) +* kubeadm: fix a possible scenario where kubeadm can pull much newer control-plane images ([#69301](https://github.com/kubernetes/kubernetes/pull/69301), [@neolit123](https://github.com/neolit123)) +* test/e2e/e2e.test: ([#69105](https://github.com/kubernetes/kubernetes/pull/69105), [@pohly](https://github.com/pohly)) + * -viper-config can be used to set also the options defined by command line flags + * the default config file is "e2e.yaml/toml/json/..." and the test starts when no such config is found (as before) but if -viper-config is used, the config file must exist + * -viper-config can be used to select a file with full path, with or without file suffix + * the csiImageVersion/Registry flags were renamed to storage.csi.imageVersion/Registry +* Move FakeCache to pkg/scheduler/internal/cache/fake. ([#69318](https://github.com/kubernetes/kubernetes/pull/69318), [@wgliang](https://github.com/wgliang)) +* The "kubectl cp" command now supports path shortcuts (../) in remote paths. ([#65189](https://github.com/kubernetes/kubernetes/pull/65189), [@juanvallejo](https://github.com/juanvallejo)) +* Fixed subpath in containerized kubelet. ([#69565](https://github.com/kubernetes/kubernetes/pull/69565), [@jsafrane](https://github.com/jsafrane)) +* The runtimeHandler field on the RuntimeClass resource now accepts the empty string. ([#69550](https://github.com/kubernetes/kubernetes/pull/69550), [@tallclair](https://github.com/tallclair)) +* Kubelet can now parse PEM file containing both TLS certificate and key in arbitrary order. Previously key was always required to be first. ([#69536](https://github.com/kubernetes/kubernetes/pull/69536), [@awly](https://github.com/awly)) +* Scheduling conformance tests related to daemonsets should set the annotation that relaxes node selection restrictions, if any are set. This ensures conformance tests can run on a wider array of clusters. ([#68793](https://github.com/kubernetes/kubernetes/pull/68793), [@aveshagarwal](https://github.com/aveshagarwal)) +* Replace Parallelize with function ParallelizeUntil and formally deprecate the Parallelize. ([#68403](https://github.com/kubernetes/kubernetes/pull/68403), [@wgliang](https://github.com/wgliang)) +* Move scheduler cache interface and implementation to pkg/scheduler/internal/cache. ([#68968](https://github.com/kubernetes/kubernetes/pull/68968), [@wgliang](https://github.com/wgliang)) +* Update to use go1.11.1 ([#69386](https://github.com/kubernetes/kubernetes/pull/69386), [@cblecker](https://github.com/cblecker)) +* Any external provider should be aware the cloud-provider interface should be imported from :- ([#68310](https://github.com/kubernetes/kubernetes/pull/68310), [@cheftako](https://github.com/cheftako)) + * cloudprovider "k8s.io/cloud-provider" +* kubeadm: Fix a crash if the etcd local alpha phase is called when the configuration contains an external etcd cluster ([#69420](https://github.com/kubernetes/kubernetes/pull/69420), [@ereslibre](https://github.com/ereslibre)) +* kubeadm now allows mixing of init/cluster and join configuration in a single YAML file (although a warning gets printed in this case). ([#69426](https://github.com/kubernetes/kubernetes/pull/69426), [@rosti](https://github.com/rosti)) +* Code-gen: Remove lowercasing for project imports ([#68484](https://github.com/kubernetes/kubernetes/pull/68484), [@jsturtevant](https://github.com/jsturtevant)) +* Fix client cert setup in delegating authentication logic ([#69430](https://github.com/kubernetes/kubernetes/pull/69430), [@DirectXMan12](https://github.com/DirectXMan12)) +* service.beta.kubernetes.io/aws-load-balancer-internal now supports true and false values, previously it only supported non-empty strings ([#69436](https://github.com/kubernetes/kubernetes/pull/69436), [@mcrute](https://github.com/mcrute)) +* OpenAPI spec and API reference now reflect dryRun query parameter for POST/PUT/PATCH operations ([#69359](https://github.com/kubernetes/kubernetes/pull/69359), [@roycaihw](https://github.com/roycaihw)) +* kubeadm: Add a `v1beta1` API. ([#69289](https://github.com/kubernetes/kubernetes/pull/69289), [@fabriziopandini](https://github.com/fabriziopandini)) +* kube-apiserver has removed support for the `etcd2` storage backend (deprecated since v1.9). Existing clusters must migrate etcd v2 data to etcd v3 storage before upgrading to v1.13. ([#69310](https://github.com/kubernetes/kubernetes/pull/69310), [@liggitt](https://github.com/liggitt)) +* List operations against the API now return internal server errors instead of partially complete lists when a value cannot be transformed from storage. The updated behavior is consistent with all other operations that require transforming data from storage such as watch and get. ([#69399](https://github.com/kubernetes/kubernetes/pull/69399), [@mikedanese](https://github.com/mikedanese)) +* `kubectl wait` now supports condition value checks other than true using `--for condition=available=false` ([#69295](https://github.com/kubernetes/kubernetes/pull/69295), [@deads2k](https://github.com/deads2k)) +* CCM server will not listen insecurely if secure port is specified ([#68982](https://github.com/kubernetes/kubernetes/pull/68982), [@aruneli](https://github.com/aruneli)) +* Bump cluster-proportional-autoscaler to 1.3.0 ([#69338](https://github.com/kubernetes/kubernetes/pull/69338), [@MrHohn](https://github.com/MrHohn)) + * - Rebase docker image on scratch. +* fix inconsistency in windows kernel proxy when updating HNS policy. ([#68923](https://github.com/kubernetes/kubernetes/pull/68923), [@delulu](https://github.com/delulu)) +* Fixes the sample-apiserver so that its BanFlunder admission plugin can be used. ([#68417](https://github.com/kubernetes/kubernetes/pull/68417), [@MikeSpreitzer](https://github.com/MikeSpreitzer)) +* Fixed CSIDriver API object to allow missing fields. ([#69331](https://github.com/kubernetes/kubernetes/pull/69331), [@jsafrane](https://github.com/jsafrane)) +* Bump addon-manager to v8.8 ([#69337](https://github.com/kubernetes/kubernetes/pull/69337), [@MrHohn](https://github.com/MrHohn)) + * - Rebase docker image on debian-base:0.3.2. +* Update defaultbackend image to 1.5. Users should concentrate on updating scripts to the new version. ([#69120](https://github.com/kubernetes/kubernetes/pull/69120), [@aledbf](https://github.com/aledbf)) +* Bump Dashboard version to v1.10.0 ([#68450](https://github.com/kubernetes/kubernetes/pull/68450), [@jeefy](https://github.com/jeefy)) +* Fixed panic on iSCSI volume tear down. ([#69140](https://github.com/kubernetes/kubernetes/pull/69140), [@jsafrane](https://github.com/jsafrane)) +* Update defaultbackend to v1.5 ([#69334](https://github.com/kubernetes/kubernetes/pull/69334), [@bowei](https://github.com/bowei)) +* Remove unused chaosclient. ([#68409](https://github.com/kubernetes/kubernetes/pull/68409), [@wgliang](https://github.com/wgliang)) +* Enable AttachVolumeLimit feature ([#69225](https://github.com/kubernetes/kubernetes/pull/69225), [@gnufied](https://github.com/gnufied)) +* Update crictl to v1.12.0 ([#69033](https://github.com/kubernetes/kubernetes/pull/69033), [@feiskyer](https://github.com/feiskyer)) +* Wait for pod failed event in subpath test. ([#69300](https://github.com/kubernetes/kubernetes/pull/69300), [@mrunalp](https://github.com/mrunalp)) +* [GCP] Added env variables to control CPU requests of kube-controller-manager and kube-scheduler. ([#68823](https://github.com/kubernetes/kubernetes/pull/68823), [@loburm](https://github.com/loburm)) +* Bump up pod short start timeout to 2 minutes. ([#69291](https://github.com/kubernetes/kubernetes/pull/69291), [@mrunalp](https://github.com/mrunalp)) +* Use the mounted "/var/run/secrets/kubernetes.io/serviceaccount/token" as the token file for running in-cluster based e2e testing. ([#69273](https://github.com/kubernetes/kubernetes/pull/69273), [@dims](https://github.com/dims)) +* apiservice availability related to networking glitches are corrected faster ([#68678](https://github.com/kubernetes/kubernetes/pull/68678), [@deads2k](https://github.com/deads2k)) +* extract volume attachment status checking operation as a common function when attaching a CSI volume ([#68931](https://github.com/kubernetes/kubernetes/pull/68931), [@mlmhl](https://github.com/mlmhl)) +* PodSecurityPolicy objects now support a `MayRunAs` rule for `fsGroup` and `supplementalGroups` options. This allows specifying ranges of allowed GIDs for pods/containers without forcing a default GID the way `MustRunAs` does. This means that a container to which such a policy applies to won't use any fsGroup/supplementalGroup GID if not explicitly specified, yet a specified GID must still fall in the GID range according to the policy. ([#65135](https://github.com/kubernetes/kubernetes/pull/65135), [@stlaz](https://github.com/stlaz)) +* Images for cloud-controller-manager, kube-apiserver, kube-controller-manager, and kube-scheduler now contain a minimal /etc/nsswitch.conf and should respect /etc/hosts for lookups ([#69238](https://github.com/kubernetes/kubernetes/pull/69238), [@BenTheElder](https://github.com/BenTheElder)) +* add deprecation warning for all cloud providers ([#69171](https://github.com/kubernetes/kubernetes/pull/69171), [@andrewsykim](https://github.com/andrewsykim)) +* IPVS proxier mode now support connection based graceful termination. ([#66012](https://github.com/kubernetes/kubernetes/pull/66012), [@Lion-Wei](https://github.com/Lion-Wei)) +* Fix panic in kubectl rollout commands ([#69150](https://github.com/kubernetes/kubernetes/pull/69150), [@soltysh](https://github.com/soltysh)) +* Add fallbacks to ARM API when getting empty node IP from Azure IMDS ([#69077](https://github.com/kubernetes/kubernetes/pull/69077), [@feiskyer](https://github.com/feiskyer)) +* Deduplicate PATH items when reading plugins. ([#69089](https://github.com/kubernetes/kubernetes/pull/69089), [@soltysh](https://github.com/soltysh)) +* Adds permissions for startup of an on-cluster kube-controller-manager ([#69062](https://github.com/kubernetes/kubernetes/pull/69062), [@dghubble](https://github.com/dghubble)) +* Fixes issue [[#68899](https://github.com/kubernetes/kubernetes/pull/68899)](https://github.com/kubernetes/kubernetes/issues/68899) where pods might schedule on an unschedulable node. ([#68984](https://github.com/kubernetes/kubernetes/pull/68984), [@k82cn](https://github.com/k82cn)) +* Returns error if NodeGetInfo fails. ([#68979](https://github.com/kubernetes/kubernetes/pull/68979), [@xing-yang](https://github.com/xing-yang)) +* Pod disruption budgets shouldn't be checked for terminal pods while evicting ([#68892](https://github.com/kubernetes/kubernetes/pull/68892), [@ravisantoshgudimetla](https://github.com/ravisantoshgudimetla)) +* Fix scheduler crashes when Prioritize Map function returns error. ([#68563](https://github.com/kubernetes/kubernetes/pull/68563), [@DylanBLE](https://github.com/DylanBLE)) +* kubeadm: create control plane with ClusterFirstWithHostNet DNS policy ([#68890](https://github.com/kubernetes/kubernetes/pull/68890), [@andrewrynhard](https://github.com/andrewrynhard)) +* Reduced excessive logging from fluentd-gcp-scaler. ([#68837](https://github.com/kubernetes/kubernetes/pull/68837), [@x13n](https://github.com/x13n)) +* adds dynamic lister ([#68748](https://github.com/kubernetes/kubernetes/pull/68748), [@p0lyn0mial](https://github.com/p0lyn0mial)) +* kubectl: add the --no-headers flag to `kubectl top ...` ([#67890](https://github.com/kubernetes/kubernetes/pull/67890), [@WanLinghao](https://github.com/WanLinghao)) +* Restrict redirect following from the apiserver to same-host redirects, and ignore redirects in some cases. ([#66516](https://github.com/kubernetes/kubernetes/pull/66516), [@tallclair](https://github.com/tallclair)) +* Fixed pod cleanup when /var/lib/kubelet is a symlink. ([#68741](https://github.com/kubernetes/kubernetes/pull/68741), [@jsafrane](https://github.com/jsafrane)) +* Add "only_cpu_and_memory" GET parameter to /stats/summary http handler in kubelet. If parameter is true then only cpu and memory will be present in response. ([#67829](https://github.com/kubernetes/kubernetes/pull/67829), [@krzysztof-jastrzebski](https://github.com/krzysztof-jastrzebski)) +* Start synchronizing pods after network is ready. ([#68752](https://github.com/kubernetes/kubernetes/pull/68752), [@krzysztof-jastrzebski](https://github.com/krzysztof-jastrzebski)) +* kubectl has gained new --profile and --profile-output options to output go profiles ([#68681](https://github.com/kubernetes/kubernetes/pull/68681), [@dlespiau](https://github.com/dlespiau)) +* Provides FSGroup capability on FlexVolume driver. It allows to disable the VolumeOwnership operation when volume is mounted ([#68680](https://github.com/kubernetes/kubernetes/pull/68680), [@benoitf](https://github.com/benoitf)) +* Apply _netdev mount option on bind mount ([#68626](https://github.com/kubernetes/kubernetes/pull/68626), [@gnufied](https://github.com/gnufied)) +* fix UnmountDevice failure on Windows ([#68608](https://github.com/kubernetes/kubernetes/pull/68608), [@andyzhangx](https://github.com/andyzhangx)) +* Allows changing nodeName in endpoint update. ([#68575](https://github.com/kubernetes/kubernetes/pull/68575), [@prameshj](https://github.com/prameshj)) +* kube-apiserver would return 400 Bad Request when it couldn't decode a json patch. ([#68346](https://github.com/kubernetes/kubernetes/pull/68346), [@CaoShuFeng](https://github.com/CaoShuFeng)) + * kube-apiserver would return 422 Unprocessable Entity when a json patch couldn't be applied to one object. +* remove unused ReplicasetControllerOptions ([#68121](https://github.com/kubernetes/kubernetes/pull/68121), [@dixudx](https://github.com/dixudx)) +* Pass signals to fluentd process ([#68064](https://github.com/kubernetes/kubernetes/pull/68064), [@gianrubio](https://github.com/gianrubio)) +* Flex drivers by default do not produce metrics. Flex plugins can enable metrics collection by setting the capability 'supportsMetrics' to true. Make sure the file system can support fs stat to produce metrics in this case. ([#67508](https://github.com/kubernetes/kubernetes/pull/67508), [@brahmaroutu](https://github.com/brahmaroutu)) +* Use monotonically increasing generation to prevent scheduler equivalence cache race. ([#67308](https://github.com/kubernetes/kubernetes/pull/67308), [@cofyc](https://github.com/cofyc)) +* Fix kubelet service file permission warning ([#66669](https://github.com/kubernetes/kubernetes/pull/66669), [@daixiang0](https://github.com/daixiang0)) +* Add prometheus metric for scheduling throughput. ([#64526](https://github.com/kubernetes/kubernetes/pull/64526), [@misterikkit](https://github.com/misterikkit)) +* Get public IP for Azure vmss nodes. ([#68498](https://github.com/kubernetes/kubernetes/pull/68498), [@feiskyer](https://github.com/feiskyer)) +* test/integration: add a basic test for covering CronJobs ([#66937](https://github.com/kubernetes/kubernetes/pull/66937), [@mortent](https://github.com/mortent)) +* Make service environment variables optional ([#68754](https://github.com/kubernetes/kubernetes/pull/68754), [@bradhoekstra](https://github.com/bradhoekstra)) diff --git a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md index ecf3ec0038..034200a1e4 100644 --- a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md +++ b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md @@ -11,13 +11,7 @@ weight: 30 {{% capture overview %}} This page explains how to add versioning information to [CustomResourceDefinitions](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#customresourcedefinition-v1beta1-apiextensions), to indicate the stability -level of your CustomResourceDefinitions. It also describes how to upgrade an -object from one version to another. - -{{< note >}} -All specified versions must use the same schema. There is no schema conversion -between versions. -{{< /note >}} +level of your CustomResourceDefinitions or advance your API to a new version with conversion between API representations. It also describes how to upgrade an object from one version to another. {{% /capture %}} @@ -36,10 +30,10 @@ between versions. ## Overview The CustomResourceDefinition API supports a `versions` field that you can use to -support multiple versions of custom resources that you have developed, and -indicate the stability of a given custom resource. All versions must currently -use the same schema, so if you need to add a field, you must add it to all -versions. +support multiple versions of custom resources that you have developed. Versions +can have different schemas with a conversion webhook to convert custom resources between versions. +Webhook conversions should follow the [Kubernetes API conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md) wherever applicable. +Specifically, See the [API change documentation](https://github.com/kubernetes/community/blob/master/contributors/devel/api_changes.md) for a set of useful gotchas and suggestions. {{< note >}} Earlier iterations included a `version` field instead of `versions`. The @@ -49,8 +43,9 @@ match the first item in the `versions` field. ## Specify multiple versions -This example shows a CustomResourceDefinition with two versions. The comments in -the YAML provide more context. +This example shows a CustomResourceDefinition with two versions. For the first +example, the assumption is all versions share the same schema with no conversion +between them. The comments in the YAML provide more context. ```yaml apiVersion: apiextensions.k8s.io/v1beta1 @@ -71,6 +66,12 @@ spec: - name: v1 served: true storage: false + # The conversion section is introduced in Kubernetes 1.13+ with a default value of + # None conversion (strategy sub-field set to None). + conversion: + # None conversion assumes the same schema for all versions and only sets the apiVersion + # field of custom resources to the proper value + strategy: None # either Namespaced or Cluster scope: Namespaced names: @@ -144,6 +145,132 @@ version sort order is `v1`, followed by `v1beta1`. This causes the kubectl command to use `v1` as the default version unless the provided object specifies the version. +## Webhook conversion + +{{< note >}} +Webhook conversion is introduced in Kubernetes 1.13 as an alpha feature. To use it, the +`CustomResourceWebhookConversion` feature should be enabled. Please refer to the [feature gate](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) documentation for more information. +{{< /note >}} + +The above example has a None conversion between versions which only sets the `apiVersion` field +on conversion and does not change the rest of the object. The API server also supports webhook +conversions that call an external service in case a conversion is required. For example when: + +* custom resource is requested in a different version than stored version. +* Watch is created in one version but the changed object is stored in another version. +* custom resource PUT request is in a different version than storage version. + +To cover all of these cases and to optimize conversion by the API server, the conversion requests may contain multiple objects in order to minimize the external calls. The webhook should perform these conversions independently. + +### Write a conversion webhook server + +Please refer to the implementation of the [custom resource conversion webhook +server](https://github.com/kubernetes/kubernetes/tree/v1.13.0/test/images/crd-conversion-webhook/main.go) +that is validated in a Kubernetes e2e test. The webhook handles the +`ConversionReview` requests sent by the API servers, and sends back conversion +results wrapped in `ConversionResponse`. Note that the request +contains a list of custom resources that need to be converted independently without +changing the order of objects. +The example server is organized in a way to be reused for other conversions. Most of the common code are located in the [framework file]((https://github.com/kubernetes/kubernetes/tree/v1.13.0/test/images/crd-conversion-webhook/converter/framework.go)) that leaves only [one function]((https://github.com/kubernetes/kubernetes/tree/v1.13.0/test/images/crd-conversion-webhook/converter/example-converter.go#L29-L80)) to be implemented for different conversions. + +{{< note >}} +The example conversion webhook server leaves the `ClientAuth` field +[empty](https://github.com/kubernetes/kubernetes/tree/v1.13.0/test/images/crd-conversion-webhook/config.go#L47-L48), +which defaults to `NoClientCert`. This means that the webhook server does not +authenticate the identity of the clients, supposedly API servers. If you need +mutual TLS or other ways to authenticate the clients, see +how to [authenticate API servers](/docs/reference/access-authn-authz/extensible-admission-controllers/#authenticate-apiservers). +{{< /note >}} + +### Deploy the conversion webhook service + +Documentation for deploying the conversion webhook is the same as for the [admission webhook example service](/docs/reference/access-authn-authz/extensible-admission-controllers/#deploy_the_admission_webhook_service). +The assumption for next sections is that the conversion webhook server is deployed to a service named `example-conversion-webhook-server` in `default` namespace. + +{{< note >}} +When the webhook server is deployed into the Kubernetes cluster as a +service, it has to be exposed via a service on port 443 (The server +itself can have an arbitrary port but the service object should map it to port 443). +The communication between the API server and the webhook service may fail +if a different port is used for the service. +{{< /note >}} + +### Configure CustomResourceDefinition to use conversion webhooks + +The `None` conversion example can be extended to use the conversion webhook by modifying `conversion` +section of the `spec`: + +```yaml +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + # name must match the spec fields below, and be in the form: . + name: crontabs.example.com +spec: + # group name to use for REST API: /apis// + group: example.com + # list of versions supported by this CustomResourceDefinition + versions: + - name: v1beta1 + # Each version can be enabled/disabled by Served flag. + served: true + # One and only one version must be marked as the storage version. + storage: true + # Each version can define it's own schema when there is no top-level + # schema is defined. + schema: + openAPIV3Schema: + properties: + hostPort: + type: string + - name: v1 + served: true + storage: false + schema: + openAPIV3Schema: + properties: + host: + type: string + port: + type: string + conversion: + # a Webhook strategy instruct API server to call an external webhook for any conversion between custom resources. + strategy: Webhook + # webhookClientConfig is required when strategy is `Webhook` and it configure the webhook endpoint to be + # called by API server. + webhookClientConfig: + service: + namespace: default + name: example-conversion-webhook-server + caBundle: + # either Namespaced or Cluster + scope: Namespaced + names: + # plural name to be used in the URL: /apis/// + plural: crontabs + # singular name to be used as an alias on the CLI and for display + singular: crontab + # kind is normally the CamelCased singular type. Your resource manifests use this. + kind: CronTab + # shortNames allow shorter string to match your resource on the CLI + shortNames: + - ct +``` + +{{< note >}} +When using `clientConfig.service`, the server cert must be valid for +`..svc`. +{{< /note >}} + +You can save the CustomResourceDefinition in a YAML file, then use +`kubectl apply` to apply it. + +```shell +kubectl apply -f my-versioned-crontab-with-conversion.yaml +``` + +Make sure the conversion service is up and running before applying new changes. + ## Writing, reading, and updating versioned CustomResourceDefinition objects When an object is written, it is persisted at the version designated as the diff --git a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md index 93478fb7d8..36ea1a0dfe 100644 --- a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md +++ b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md @@ -146,10 +146,8 @@ items: - apiVersion: stable.example.com/v1 kind: CronTab metadata: - clusterName: "" creationTimestamp: 2017-05-31T12:56:35Z - deletionGracePeriodSeconds: null - deletionTimestamp: null + generation: 1 name: my-new-cron-object namespace: default resourceVersion: "285" @@ -175,7 +173,7 @@ kubectl get crontabs ``` ```console -Error from server (NotFound): Unable to list "crontabs": the server could not find the requested resource (get crontabs.stable.example.com) +Error from server (NotFound): Unable to list {"stable.example.com" "v1" "crontabs"}: the server could not find the requested resource (get crontabs.stable.example.com) ``` If you later recreate the same CustomResourceDefinition, it will start out empty. @@ -470,7 +468,7 @@ When the status subresource is enabled, the `/status` subresource for the custom - `PUT` requests to the `/status` subresource take a custom resource object and ignore changes to anything except the status stanza. - `PUT` requests to the `/status` subresource only validate the status stanza of the custom resource. - `PUT`/`POST`/`PATCH` requests to the custom resource ignore changes to the status stanza. -- Any changes to the spec stanza increments the value at `.metadata.generation`. +- The `.metadata.generation` value is incremented for all changes, except for changes to `.metadata` or `.status`. - Only the following constructs are allowed at the root of the CRD OpenAPI validation schema: - Description diff --git a/content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md b/content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md index 0e5a553571..e00fda711d 100644 --- a/content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md +++ b/content/en/docs/tasks/administer-cluster/configure-upgrade-etcd.md @@ -201,223 +201,29 @@ If the majority of etcd members have permanently failed, the etcd cluster is con ## Upgrading and rolling back etcd clusters -### Important assumptions +As of Kubernetes v1.13.0, etcd2 is no longer supported as a storage backend for +new or existing Kubernetes clusters. The timeline for Kubernetes support for +etcd2 and etcd3 is as follows: -The upgrade procedure described in this document assumes that either: +- Kubernetes v1.0: etcd2 only +- Kubernetes v1.5.1: etcd3 support added, new clusters still default to etcd2 +- Kubernetes v1.6.0: new clusters created with `kube-up.sh` default to etcd3, + and `kube-apiserver` defaults to etcd3 +- Kubernetes v1.9.0: deprecation of etcd2 storage backend announced +- Kubernetes v1.13.0: etcd2 storage backend removed, `kube-apiserver` will + refuse to start with `--storage-backend=etcd2`, with the + message `etcd2 is no longer a supported storage backend` -1. The etcd cluster has only a single node. -2. The etcd cluster has multiple nodes. +Before upgrading a v1.12.x kube-apiserver using `--storage-backend=etcd2` to +v1.13.x, etcd v2 data MUST by migrated to the v3 storage backend, and +kube-apiserver invocations changed to use `--storage-backend=etcd3`. - In this case, the upgrade procedure requires shutting down the - etcd cluster. During the time the etcd cluster is shut down, the Kubernetes API Server will be read only. +The process for migrating from etcd2 to etcd3 is highly dependent on how the +etcd cluster was deployed and configured, as well as how the Kubernetes +cluster was deployed and configured. We recommend that you consult your cluster +provider's documentation to see if there is a predefined solution. -{{< warning >}} -Deviations from the assumptions are untested by continuous -integration, and deviations might create undesirable consequences. Additional information about operating an etcd cluster is available [from the etcd maintainers](https://github.com/coreos/etcd/tree/master/Documentation). -{{< /warning >}} - -### Background - -As of Kubernetes version 1.5.1, we are still using etcd from the 2.2.1 release with -the v2 API. Also, we have no pre-existing process for updating etcd, as we have -never updated etcd by either minor or major version. - -Note that we need to migrate both the etcd versions that we are using (from 2.2.1 -to at least 3.0.x) as well as the version of the etcd API that Kubernetes talks to. The etcd 3.0.x -binaries support both the v2 and v3 API. - -This document describes how to do this migration. If you want to skip the -background and cut right to the procedure, see [Upgrade -Procedure](#upgrade-procedure). - -### etcd upgrade requirements - -There are requirements on how an etcd cluster upgrade can be performed. The primary considerations are: -- Upgrade between one minor release at a time -- Rollback supported through additional tooling - -#### One minor release at a time - -Upgrade only one minor release at a time. For example, we cannot upgrade directly from 2.1.x to 2.3.x. -Within patch releases it is possible to upgrade and downgrade between arbitrary versions. Starting a cluster for -any intermediate minor release, waiting until the cluster is healthy, and then -shutting down the cluster will perform the migration. For example, to upgrade from version 2.1.x to 2.3.y, -it is enough to start etcd in 2.2.z version, wait until it is healthy, stop it, and then start the -2.3.y version. - -#### Rollback via additional tooling - -Versions 3.0+ of etcd do not support general rollback. That is, -after migrating from M.N to M.N+1, there is no way to go back to M.N. -The etcd team has provided a [custom rollback tool](https://git.k8s.io/kubernetes/cluster/images/etcd/rollback) -but the rollback tool has these limitations: - -* This custom rollback tool is not part of the etcd repo and does not receive the same - testing as the rest of etcd. We are testing it in a couple of end-to-end tests. - There is only community support here. - -* The rollback can be done only from the 3.0.x version (that is using the v3 API) to the - 2.2.1 version (that is using the v2 API). - -* The tool only works if the data is stored in `application/json` format. - -* Rollback doesn’t preserve resource versions of objects stored in etcd. - -{{< warning >}} -If the data is not kept in `application/json` format (see [Upgrade -Procedure](#upgrade-procedure)), you will lose the option to roll back to etcd -2.2. -{{< /warning >}} - -The last bullet means that any component or user that has some logic -depending on resource versions may require restart after etcd rollback. This -includes that all clients using the watch API, which depends on -resource versions. Since both the kubelet and kube-proxy use the watch API, a -rollback might require restarting all Kubernetes components on all nodes. - -{{< note >}} -At the time of writing, both Kubelet and KubeProxy are using “resource -version” only for watching (i.e. are not using resource versions for anything -else). And both are using reflector and/or informer frameworks for watching -(i.e. they don’t send watch requests themselves). Both those frameworks if they -can’t renew watch, they will start from “current version” by doing “list + watch -from the resource version returned by list”. That means that if the apiserver -will be down for the period of rollback, all of node components should basically -restart their watches and start from “now” when apiserver is back. And it will -be back with new resource version. That would mean that restarting node -components is not needed. But the assumptions here may not hold forever. -{{< /note >}} - -{{% /capture %}} - -{{% capture discussion %}} - -### Design - -This section describes how we are going to do the migration, given the -[etcd upgrade requirements](#etcd-upgrade-requirements). - -Note that because the code changes in Kubernetes code needed -to support the etcd v3 API are local and straightforward, we do not -focus on them at all. We focus only on the upgrade/rollback here. - -### New etcd Docker image - -We decided to completely change the content of the etcd image and the way it works. -So far, the Docker image for etcd in version X has contained only the etcd and -etcdctl binaries. - -Going forward, the Docker image for etcd in version X will contain multiple -versions of etcd. For example, the 3.0.17 image will contain the 2.2.1, 2.3.7, and -3.0.17 binaries of etcd and etcdctl. This will allow running etcd in multiple -different versions using the same Docker image. - -Additionally, the image will contain a custom script, written by the Kubernetes team, -for doing migration between versions. The image will also contain the rollback tool -provided by the etcd team. - -### Migration script -The migration script that will be part of the etcd Docker image is a bash -script that works as follows: - -1. Detect which version of etcd we were previously running. - For that purpose, we have added a dedicated file, `version.txt`, that - holds that information and is stored in the etcd-data-specific directory, - next to the etcd data. If the file doesn’t exist, we default it to version 2.2.1. -1. If we are in version 2.2.1 and are supposed to upgrade, backup - data. -1. Based on the detected previous etcd version and the desired one - (communicated via environment variable), do the upgrade steps as - needed. This means that for every minor etcd release greater than the detected one and - less than or equal to the desired one: - 1. Start etcd in that version. - 1. Wait until it is healthy. Healthy means that you can write some data to it. - 1. Stop this etcd. Note that this etcd will not listen on the default - etcd port. It is hard coded to listen on ports that the API server is not - configured to connect to, which means that API server won’t be able to connect - to it. Assuming no other client goes out of its way to try to - connect and write to this obscure port, no new data will be written during - this period. -1. If the desired API version is v3 and the detected version is v2, do the offline - migration from the v2 to v3 data format. For that we use two tools: - * ./etcdctl migrate: This is the official tool for migration provided by the etcd team. - * A custom script that is attaching TTLs to events in the etcd. Note that etcdctl - migrate doesn’t support TTLs. -1. After every successful step, update contents of the version file. - This will protect us from the situation where something crashes in the - meantime ,and the version file gets completely unsynchronized with the - real data. Note that it is safe if the script crashes after the step is - done and before the file is updated. This will only result in redoing one - step in the next try. - -All the previous steps are for the case where the detected version is less than or -equal to the desired version. In the opposite case, that is for a rollback, the -script works as follows: - -1. Verify that the detected version is 3.0.x with the v3 API, and the - desired version is 2.2.1 with the v2 API. We don’t support any other rollback. -1. If so, we run the custom tool provided by etcd team to do the offline - rollback. This tool reads the v3 formatted data and writes it back to disk - in v2 format. -1. Finally update the contents of the version file. - -### Upgrade procedure -Simply modify the command line in the etcd manifest to: - -1. Run the migration script. If the previously run version is already in the - desired version, this will be no-op. -1. Start etcd in the desired version. - -Starting in Kubernetes version 1.6, this has been done in the manifests for new -Google Compute Engine clusters. You should also specify these environment -variables. In particular, you must keep `STORAGE_MEDIA_TYPE` set to -`application/json` if you wish to preserve the option to roll back. - -``` -TARGET_STORAGE=etcd3 -ETCD_IMAGE=3.0.17 -TARGET_VERSION=3.0.17 -STORAGE_MEDIA_TYPE=application/json -``` - -To roll back, use these: - -``` -TARGET_STORAGE=etcd2 -ETCD_IMAGE=3.0.17 -TARGET_VERSION=2.2.1 -STORAGE_MEDIA_TYPE=application/json -``` - -{{< note >}} -This procedure upgrades from 2.x to 3.x. Version `3.0.17` is not recommended for running in production (see [prerequisites](#prerequisites) for minimum recommended etcd versions). -{{< /note >}} - -## Notes for etcd Version 2.2.1 - -### Default configuration - -The default setup scripts use kubelet's file-based static pods feature to run etcd in a -[pod](http://releases.k8s.io/{{< param "githubbranch" >}}/cluster/gce/manifests/etcd.manifest). This manifest should only -be run on master VMs. The default location that kubelet scans for manifests is -`/etc/kubernetes/manifests/`. - -### Kubernetes's usage of etcd - -By default, Kubernetes objects are stored under the `/registry` key in etcd. -This path can be prefixed by using the [kube-apiserver](/docs/admin/kube-apiserver) flag -`--etcd-prefix="/foo"`. - -`etcd` is the only place that Kubernetes keeps state. - -### Troubleshooting - -To test whether `etcd` is running correctly, you can try writing a value to a -test key. On your master VM (or somewhere with firewalls configured such that -you can talk to your cluster's etcd), try: - -```shell -curl -X PUT "http://${host}:${port}/v2/keys/_test" -``` +If your cluster was created via `kube-up.sh` and is still using etcd2 as its +storage backend, please consult the [Kubernetes v1.12 etcd cluster upgrade docs](https://v1-12.docs.kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#upgrading-and-rolling-back-etcd-clusters) {{% /capture %}} diff --git a/content/en/docs/tasks/administer-cluster/coredns.md b/content/en/docs/tasks/administer-cluster/coredns.md index db6d1881f6..a892cc69c7 100644 --- a/content/en/docs/tasks/administer-cluster/coredns.md +++ b/content/en/docs/tasks/administer-cluster/coredns.md @@ -42,6 +42,10 @@ during an upgrade. For example, here is what a `v1.11.0` upgrade would look like kubeadm upgrade apply v1.11.0 --feature-gates=CoreDNS=true ``` +In Kubernetes version 1.13 and later the `CoreDNS` feature gate is removed and CoreDNS +is used by default. Follow the guide outlined [here](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase#cmd-phase-addon) if you want +your upgraded cluster to use kube-dns. + In versions prior to 1.11 the Corefile will be **overwritten** by the one created during upgrade. **You should save your existing ConfigMap if you have customized it.** You may re-apply your customizations after the new ConfigMap is up and running. @@ -56,13 +60,15 @@ In Kubernetes 1.11, CoreDNS has graduated to General Availability (GA) and is installed by default. {{< /note >}} -To install kube-dns instead, set the `CoreDNS` feature gate +To install kube-dns on versions prior to 1.13, set the `CoreDNS` feature gate value to `false`: ``` kubeadm init --feature-gates=CoreDNS=false ``` +For versions 1.13 and later, follow the guide outlined [here](/docs/reference/setup-tools/kubeadm/kubeadm-init-phase#cmd-phase-addon). + ## Tuning CoreDNS When resource utilisation is a concern, it may be useful to tune the configuration of CoreDNS. For more details, check out the diff --git a/content/en/docs/tasks/administer-cluster/encrypt-data.md b/content/en/docs/tasks/administer-cluster/encrypt-data.md index 405b1aa96f..2ff37c188d 100644 --- a/content/en/docs/tasks/administer-cluster/encrypt-data.md +++ b/content/en/docs/tasks/administer-cluster/encrypt-data.md @@ -13,7 +13,7 @@ This page shows how to enable and configure encryption of secret data at rest. * {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} -* Kubernetes version 1.7.0 or later is required +* Kubernetes version 1.13.0 or later is required * etcd v3 or later is required @@ -23,15 +23,18 @@ This page shows how to enable and configure encryption of secret data at rest. ## Configuration and determining whether encryption at rest is already enabled -The `kube-apiserver` process accepts an argument `--experimental-encryption-provider-config` +The `kube-apiserver` process accepts an argument `--encryption-provider-config` that controls how API data is encrypted in etcd. An example configuration is provided below. +Note: +The alpha version of the encryption feature prior to 1.13 used the `--experimental-encryption-provider-config` flag. + ## Understanding the encryption at rest configuration. ```yaml -kind: EncryptionConfig -apiVersion: v1 +kind: EncryptionConfiguration +apiVersion: apiserver.config.k8s.io/v1 resources: - resources: - secrets @@ -66,8 +69,12 @@ resources from storage each provider that matches the stored data attempts to de order. If no provider can read the stored data due to a mismatch in format or secret key, an error is returned which prevents clients from accessing that resource. +Note: +The alpha version of the encryption feature prior to 1.13 required to be configured with +`kind: EncryptionConfig` and `apiVersion: v1`. + {{< caution >}} -If any resource is not readable via the encryption config (because keys were changed), +**IMPORTANT:** If any resource is not readable via the encryption config (because keys were changed), the only recourse is to delete that key from the underlying etcd directly. Calls that attempt to read that resource will fail until it is deleted or a valid decryption key is provided. {{< /caution >}} @@ -90,8 +97,8 @@ is the first provider, the first key is used for encryption. Create a new encryption config file: ```yaml -kind: EncryptionConfig -apiVersion: v1 +kind: EncryptionConfiguration +apiVersion: apiserver.config.k8s.io/v1 resources: - resources: - secrets @@ -112,7 +119,7 @@ To create a new secret perform the following steps: ``` 2. Place that value in the secret field. -3. Set the `--experimental-encryption-provider-config` flag on the `kube-apiserver` to point to the location of the config file. +3. Set the `--encryption-provider-config` flag on the `kube-apiserver` to point to the location of the config file. 4. Restart your API server. {{< caution >}} @@ -183,8 +190,8 @@ With a single `kube-apiserver`, step 2 may be skipped. To disable encryption at rest place the `identity` provider as the first entry in the config: ```yaml -kind: EncryptionConfig -apiVersion: v1 +kind: EncryptionConfiguration +apiVersion: apiserver.config.k8s.io/v1 resources: - resources: - secrets diff --git a/content/en/docs/tasks/administer-cluster/kms-provider.md b/content/en/docs/tasks/administer-cluster/kms-provider.md index 5f14375b9e..601b0fb97d 100644 --- a/content/en/docs/tasks/administer-cluster/kms-provider.md +++ b/content/en/docs/tasks/administer-cluster/kms-provider.md @@ -79,8 +79,8 @@ To encrypt the data: 1. Create a new encryption configuration file using the appropriate properties for the `kms` provider: ```yaml -kind: EncryptionConfig -apiVersion: v1 +kind: EncryptionConfiguration +apiVersion: apiserver.config.k8s.io/v1 resources: - resources: - secrets @@ -92,9 +92,13 @@ resources: - identity: {} ``` -2. Set the `--experimental-encryption-provider-config` flag on the kube-apiserver to point to the location of the configuration file. +2. Set the `--encryption-provider-config` flag on the kube-apiserver to point to the location of the configuration file. 3. Restart your API server. +Note: +The alpha version of the encryption feature prior to 1.13 required a config file with +`kind: EncryptionConfig` and `apiVersion: v1`, and used the `--experimental-encryption-provider-config` flag. + ## Verifying that the data is encrypted Data is encrypted when written to etcd. After restarting your kube-apiserver, any newly created or updated secret should be encrypted when stored. To verify, you can use the etcdctl command line program to retrieve the contents of your secret. @@ -130,8 +134,8 @@ To switch from a local encryption provider to the `kms` provider and re-encrypt 1. Add the `kms` provider as the first entry in the configuration file as shown in the following example. ```yaml -kind: EncryptionConfig -apiVersion: v1 +kind: EncryptionConfiguration +apiVersion: apiserver.config.k8s.io/v1 resources: - resources: - secrets @@ -160,8 +164,8 @@ To disable encryption at rest: 1. Place the `identity` provider as the first entry in the configuration file: ```yaml -kind: EncryptionConfig -apiVersion: v1 +kind: EncryptionConfiguration +apiVersion: apiserver.config.k8s.io/v1 resources: - resources: - secrets diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md new file mode 100644 index 0000000000..6414a9e8de --- /dev/null +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md @@ -0,0 +1,124 @@ +--- +reviewers: +- sig-cluster-lifecycle +title: Certificate Management with kubeadm +content_template: templates/task +--- + +{{% capture overview %}} + +This page explains how to manage certificates manually with kubeadm. + +{{% /capture %}} + +{{% capture prerequisites %}} + +These are advanced topics for users who need to integrate their organization's certificate infrastructure into a kubeadm-built cluster. If kubeadm with the default configuration satisfies your needs, you should let kubeadm manage certificates instead. + +You should be familiar with [PKI certificates and requirements in Kubernetes](/docs/setup/certificates/). + +{{% /capture %}} + +{{% capture steps %}} + +## Renew certificates with the certificates API + +Kubeadm can renew certificates with the `kubeadm alpha certs renew` commands. + +Typically this is done by loading on-disk CA certificates and keys and using them to issue new certificates. +This approach works well if your certificate tree is self-contained. However, if your certificates are externally +managed, you might need a different approach. + +As an alternative, Kubernetes provides its own [API for managing certificates][manage-tls]. +With kubeadm, you can use this API by running `kubeadm alpha certs renew --use-api`. + +## Set up a signer + +The Kubernetes Certificate Authority does not work out of the box. +You can configure an external signer such as [cert-manager][cert-manager-issuer], or you can use the build-in signer. +The built-in signer is part of [`kube-controller-manager`][kcm]. +To activate the build-in signer, you pass the `--cluster-signing-cert-file` and `--cluster-signing-key-file` arguments. + +You pass these arguments in any of the following ways: + +* Edit `/etc/kubernetes/manifests/kube-controller-manager.yaml` to add the arguments to the command. + Remember that your changes could be overwritten when you upgrade. + +* If you're creating a new cluster, you can use a kubeadm [configuration file][config]: + + ```yaml + apiVersion: kubeadm.k8s.io/v1beta1 + kind: ClusterConfiguration + controllerManager: + extraArgs: + cluster-signing-cert-file: /etc/kubernetes/pki/ca.crt + cluster-signing-key-file: /etc/kubernetes/pki/ca.key + ``` + +* You can also upload a config file using [`kubeadm config upload from-files`][config-upload] + +[cert-manager-issuer]: https://cert-manager.readthedocs.io/en/latest/tutorials/ca/creating-ca-issuer.html +[kcm]: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/ +[config]: https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1 +[config-upload]: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-config/#cmd-config-from-file + +### Approve requests + +If you set up an external signer such as [cert-manager][cert-manager], certificate signing requests (CSRs) are automatically approved. +Otherwise, you must manually approve certificates with the [`kubectl certificates`][certs] command. +The following kubeadm command outputs the name of the certificate to approve, then blocks and waits for approval to occur: + +```shell +$ sudo kubeadm alpha certs renew apiserver --use-api & +[1] 2890 +[certs] certificate request "kubeadm-cert-kube-apiserver-ld526" created +$ kubectl certificate approve kubeadm-cert-kube-apiserver-ld526 +certificatesigningrequest.certificates.k8s.io/kubeadm-cert-kube-apiserver-ld526 approved +[1]+ Done sudo kubeadm alpha certs renew apiserver --use-api +``` + +You can view a list of pending certificates with `kubectl get csr`. + +[manage-tls]: https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/ +[cert-manager]: https://github.com/jetstack/cert-manager +[certs]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#certificate + +## Certificate requests with kubeadm + +To better integrate with external CAs, kubeadm can also produce certificate signing requests (CSRs). +A CSR represents a request to a CA for a signed certificate for a client. +In kubeadm terms, any certificate that would normally be signed by an on-disk CA can be produced as a CSR instead. A CA, however, cannot be produced as a CSR. + +You can create an individual CSR with `kubeadm init phase certs apiserver --use-csr`. +The `--use-csr` flag can be applied only to individual phases. After [all certificates are in place][certs], you can run `kubeadm init --external-ca`. + +You can pass in a directory with `--csr-dir` to output the CSRs to the specified location. +If `--csr-dire` is not specified, the default certificate directory (`/etc/kubernetes/pki`) is used. +Both the CSR and the accompanying private key are given in the output. After a certificate is signed, the certificate and the private key must be copied to the PKI directory (by default `/etc/kubernetes/pki`). + +### Renew certificates + +Certificates can be renewed with `kubeadm alpha certs renew --use-csr`. +As with `kubeadm init`, an output directory can be specified with the `--csr-dir` flag. +To use the new certificates, copy the signed certificate and private key into the PKI directory (by default `/etc/kubernetes/pki`) + +## Cert usage + +A CSR contains a certificate's name, domains, and IPs, but it does not specify usages. +It is the responsibility of the CA to specify [the correct cert usages][cert-table] when issuing a certificate. + +* In `openssl` this is done with the [`openssl ca` command][openssl-ca]. +* In `cfssl` you specify [usages in the config file][cfssl-usages] + +## CA selection + +Kubeadm sets up [three CAs][cert-cas] by default. Make sure to sign the CSRs with a corresponding CA. + +[openssl-ca]: https://superuser.com/questions/738612/openssl-ca-keyusage-extension +[cfssl-usages]: https://github.com/cloudflare/cfssl/blob/master/doc/cmd/cfssl.txt#L170 +[certs]: https://kubernetes.io/docs/setup/certificates +[cert-cas]: https://kubernetes.io/docs/setup/certificates/#single-root-ca +[cert-table]: https://kubernetes.io/docs/setup/certificates/#all-certificates + +{{% /capture %}} +https://prow.k8s.io/?pull=71212 diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13.md new file mode 100644 index 0000000000..728f88936c --- /dev/null +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13.md @@ -0,0 +1,278 @@ +--- +reviewers: +- sig-cluster-lifecycle +title: Upgrading kubeadm clusters from v1.12 to v1.13 +content_template: templates/task +--- + +{{% capture overview %}} + +This page explains how to upgrade a Kubernetes cluster created with `kubeadm` from version 1.12.x to version 1.13.x, and from version 1.13.x to 1.13.y, where `y > x`. + +{{% /capture %}} + +{{% capture prerequisites %}} + +- You need to have a `kubeadm` Kubernetes cluster running version 1.12.0 or later. + [Swap must be disabled][swap]. + The cluster should use a static control plane and etcd pods. +- Make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md) carefully. +- Make sure to back up any important components, such as app-level state stored in a database. + `kubeadm upgrade` does not touch your workloads, only components internal to Kubernetes, but backups are always a best practice. + + +[swap]: https://serverfault.com/questions/684771/best-way-to-disable-swap-in-linux +### Additional information + +- All containers are restarted after upgrade, because the container spec hash value is changed. +- You can upgrade only from one minor version to the next minor version. + That is, you cannot skip versions when you upgrade. + For example, you can upgrade only from 1.10 to 1.11, not from 1.9 to 1.11. + +{{% /capture %}} + +{{% capture steps %}} + +## Upgrade the control plane + +1. On your master node, upgrade kubeadm: + + {{< tabs name="k8s_install" >}} + {{% tab name="Ubuntu, Debian or HypriotOS" %}} + apt-get update + apt-get upgrade -y kubelet kubeadm + {{% /tab %}} + {{% tab name="CentOS, RHEL or Fedora" %}} + yum upgrade -y kubeadm --disableexcludes=kubernetes + {{% /tab %}} + {{< /tabs >}} + +1. Verify that the download works and has the expected version: + + ```shell + kubeadm version + ``` + +1. On the master node, run: + + ```shell + kubeadm upgrade plan + ``` + + You should see output similar to this: + + ```shell + [preflight] Running pre-flight checks. + [upgrade] Making sure the cluster is healthy: + [upgrade/config] Making sure the configuration is correct: + [upgrade/config] Reading configuration from the cluster... + [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' + [upgrade] Fetching available versions to upgrade to + [upgrade/versions] Cluster version: v1.12.2 + [upgrade/versions] kubeadm version: v1.13.0 + + Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': + COMPONENT CURRENT AVAILABLE + Kubelet 2 x v1.12.2 v1.13.0 + + Upgrade to the latest version in the v1.12 series: + + COMPONENT CURRENT AVAILABLE + API Server v1.12.2 v1.13.0 + Controller Manager v1.12.2 v1.13.0 + Scheduler v1.12.2 v1.13.0 + Kube Proxy v1.12.2 v1.13.0 + CoreDNS 1.2.2 1.2.6 + Etcd 3.2.24 3.2.24 + + You can now apply the upgrade by executing the following command: + + kubeadm upgrade apply v1.13.0 + + _____________________________________________________________________ + ``` + + This command checks that your cluster can be upgraded, and fetches the versions you can upgrade to. + +1. Choose a version to upgrade to, and run the appropriate command. For example: + + ```shell + kubeadm upgrade apply v1.13.0 + ``` + + You should see output similar to this: + + + + ```shell + [preflight] Running pre-flight checks. + [upgrade] Making sure the cluster is healthy: + [upgrade/config] Making sure the configuration is correct: + [upgrade/config] Reading configuration from the cluster... + [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' + [upgrade/apply] Respecting the --cri-socket flag that is set with higher priority than the config file. + [upgrade/version] You have chosen to change the cluster version to "v1.13.0" + [upgrade/versions] Cluster version: v1.12.2 + [upgrade/versions] kubeadm version: v1.13.0 + [upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y + [upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd] + [upgrade/prepull] Prepulling image for component etcd. + [upgrade/prepull] Prepulling image for component kube-controller-manager. + [upgrade/prepull] Prepulling image for component kube-scheduler. + [upgrade/prepull] Prepulling image for component kube-apiserver. + [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager + [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd + [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler + [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver + [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager + [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd + [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler + [upgrade/prepull] Prepulled image for component etcd. + [upgrade/prepull] Prepulled image for component kube-apiserver. + [upgrade/prepull] Prepulled image for component kube-scheduler. + [upgrade/prepull] Prepulled image for component kube-controller-manager. + [upgrade/prepull] Successfully prepulled the images for all the control plane components + [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.13.0"... + Static pod: kube-apiserver-ip-10-0-0-7 hash: 4af3463d6ace12615f1795e40811c1a1 + Static pod: kube-controller-manager-ip-10-0-0-7 hash: a640b0098f5bddc701786e007c96e220 + Static pod: kube-scheduler-ip-10-0-0-7 hash: ee7b1077c61516320f4273309e9b4690 + map[localhost:2379:3.2.24] + [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests969681047" + [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2018-11-20-18-30-42/kube-apiserver.yaml" + [upgrade/staticpods] Waiting for the kubelet to restart the component + [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) + Static pod: kube-apiserver-ip-10-0-0-7 hash: 4af3463d6ace12615f1795e40811c1a1 + Static pod: kube-apiserver-ip-10-0-0-7 hash: bf5b045d2be93e73654f3eb7027a4ef8 + [apiclient] Found 1 Pods for label selector component=kube-apiserver + [upgrade/staticpods] Component "kube-apiserver" upgraded successfully! + [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2018-11-20-18-30-42/kube-controller-manager.yaml" + [upgrade/staticpods] Waiting for the kubelet to restart the component + [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) + Static pod: kube-controller-manager-ip-10-0-0-7 hash: a640b0098f5bddc701786e007c96e220 + Static pod: kube-controller-manager-ip-10-0-0-7 hash: 1e0eea23b3d971460ac032c18ab7daac + [apiclient] Found 1 Pods for label selector component=kube-controller-manager + [upgrade/staticpods] Component "kube-controller-manager" upgraded successfully! + [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2018-11-20-18-30-42/kube-scheduler.yaml" + [upgrade/staticpods] Waiting for the kubelet to restart the component + [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) + Static pod: kube-scheduler-ip-10-0-0-7 hash: ee7b1077c61516320f4273309e9b4690 + Static pod: kube-scheduler-ip-10-0-0-7 hash: 7f7d929b61a2cc5bcdf36609f75927ec + [apiclient] Found 1 Pods for label selector component=kube-scheduler + [upgrade/staticpods] Component "kube-scheduler" upgraded successfully! + [uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace + [kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster + [kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace + [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" + [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "ip-10-0-0-7" as an annotation + [bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials + [bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token + [bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster + [addons] Applied essential addon: CoreDNS + [addons] Applied essential addon: kube-proxy + + [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.13.0". Enjoy! + + [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so. + ``` + +1. Manually upgrade your Software Defined Network (SDN). + + Your Container Network Interface (CNI) provider may have its own upgrade instructions to follow. + Check the [addons](/docs/concepts/cluster-administration/addons/) page to + find your CNI provider and see whether additional upgrade steps are required. + +## Upgrade master and node packages + +1. Prepare each node for maintenance by marking it unschedulable and evicting the workloads. Run: + + ```shell + kubectl drain $NODE --ignore-daemonsets + ``` + + On the master node, you must add `--ignore-daemonsets`: + + ```shell + kubectl drain ip-172-31-85-18 + node "ip-172-31-85-18" cordoned + error: unable to drain node "ip-172-31-85-18", aborting command... + + There are pending nodes to be drained: + ip-172-31-85-18 + error: DaemonSet-managed pods (use --ignore-daemonsets to ignore): calico-node-5798d, kube-proxy-thjp9 + ``` + + ``` + kubectl drain ip-172-31-85-18 --ignore-daemonsets + node "ip-172-31-85-18" already cordoned + WARNING: Ignoring DaemonSet-managed pods: calico-node-5798d, kube-proxy-thjp9 + node "ip-172-31-85-18" drained + ``` + +1. Upgrade the Kubernetes package version on each `$NODE` node by running the Linux package manager for your distribution: + + {{< tabs name="k8s_install" >}} + {{% tab name="Ubuntu, Debian or HypriotOS" %}} + apt-get update + apt-get upgrade -y kubelet kubeadm + {{% /tab %}} + {{% tab name="CentOS, RHEL or Fedora" %}} + yum upgrade -y kubelet kubeadm --disableexcludes=kubernetes + {{% /tab %}} + {{< /tabs >}} + +## Upgrade kubelet on each node + +1. On each node except the master node, upgrade the kubelet config: + + ```shell + kubeadm upgrade node config --kubelet-version $(kubelet --version | cut -d ' ' -f 2) + ``` + +1. Restart the kubelet process: + + ```shell + systemctl restart kubelet + ``` + +1. Verify that the new version of the `kubelet` is running on the node: + + ```shell + systemctl status kubelet + ``` + +1. Bring the node back online by marking it schedulable: + + ```shell + kubectl uncordon $NODE + ``` + +1. After the kubelet is upgraded on all nodes, verify that all nodes are available again by running the following command from anywhere kubectl can access the cluster: + + ```shell + kubectl get nodes + ``` + + The `STATUS` column should show `Ready` for all your nodes, and the version number should be updated. + +{{% /capture %}} + +## Recovering from a failure state + +If `kubeadm upgrade` fails and does not roll back, for example because of an unexpected shutdown during execution, you can run `kubeadm upgrade` again. +This command is idempotent and eventually makes sure that the actual state is the desired state you declare. + +To recover from a bad state, you can also run `kubeadm upgrade --force` without changing the version that your cluster is running. + +## How it works + +`kubeadm upgrade apply` does the following: + +- Checks that your cluster is in an upgradeable state: + - The API server is reachable + - All nodes are in the `Ready` state + - The control plane is healthy +- Enforces the version skew policies. +- Makes sure the control plane images are available or available to pull to the machine. +- Upgrades the control plane components or rollbacks if any of them fails to come up. +- Applies the new `kube-dns` and `kube-proxy` manifests and makes sure that all necessary RBAC rules are created. +- Creates new certificate and key files of the API server and backs up old files if they're about to expire in 180 days. diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha-1-12.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha-1-12.md index 860ff92cdf..f43cd16567 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha-1-12.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha-1-12.md @@ -113,7 +113,7 @@ You should see something like the following: [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.12.0". Enjoy! -The `kubeadm-config` ConfigMap is now updated from `v1alpha2` version to `v1alpha3`. +The `kubeadm-config` ConfigMap is now updated from `v1alpha3` version to `v1beta1`. ### Upgrading additional control plane nodes diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha-1-13.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha-1-13.md new file mode 100644 index 0000000000..8b019ea043 --- /dev/null +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha-1-13.md @@ -0,0 +1,168 @@ +--- +reviewers: +- luxas +- timothysc +- jbeda +title: Upgrading kubeadm HA clusters from v1.12 to v1.13 +content_template: templates/task +--- + +{{% capture overview %}} + +This page explains how to upgrade a highly available (HA) Kubernetes cluster created with `kubeadm` from version 1.12.x to version 1.13.y. In addition to upgrading, you must also follow the instructions in [Creating HA clusters with kubeadm](/docs/setup/independent/high-availability/). + +{{% /capture %}} + +{{% capture prerequisites %}} + +Before proceeding: + +- You need to have a `kubeadm` HA cluster running version 1.12 or higher. +- Make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md) carefully. +- Make sure to back up any important components, such as app-level state stored in a database. `kubeadm upgrade` does not touch your workloads, only components internal to Kubernetes, but backups are always a best practice. +- Check the prerequisites for [Upgrading/downgrading kubeadm clusters between v1.12 to v1.13](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/). + +{{< note >}} +All commands on any control plane or etcd node should be run as root. +{{< /note >}} + +{{% /capture %}} + +{{% capture steps %}} + +## Prepare for both methods + +Upgrade `kubeadm` to the version that matches the version of Kubernetes that you are upgrading to: + +```shell +apt-mark unhold kubeadm && \ +apt-get update && apt-get install -y kubeadm && \ +apt-mark hold kubeadm +``` + +Check prerequisites and determine the upgrade versions: + +```shell +kubeadm upgrade plan +``` + +You should see something like the following: + +``` +Upgrade to the latest version in the v1.13 series: + +COMPONENT CURRENT AVAILABLE +API Server v1.12.2 v1.13.0 +Controller Manager v1.12.2 v1.13.0 +Scheduler v1.12.2 v1.13.0 +Kube Proxy v1.12.2 v1.13.0 +CoreDNS 1.2.2 1.2.6 +``` + +## Stacked control plane nodes + +### Upgrade the first control plane node + +Modify `configmap/kubeadm-config` for this control plane node: + +```shell +kubectl edit configmap -n kube-system kubeadm-config +``` + +Make the following modifications to the ClusterConfiguration key: + +- `etcd` + + Remove the etcd section completely + +Make the following modifications to the ClusterStatus key: + +- `apiEndpoints` + + Add an entry for each of the additional control plane hosts + +Start the upgrade: + +```shell +kubeadm upgrade apply v +``` + +You should see something like the following: + + [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.13.0". Enjoy! + +The `kubeadm-config` ConfigMap is now updated from `v1alpha3` version to `v1beta1`. + +### Upgrading additional control plane nodes + +Start the upgrade: + +```shell +kubeadm upgrade node experimental-control-plane +``` + +## External etcd + +### Upgrade the first control plane + +Run the upgrade: + +``` +kubeadm upgrade apply v1.13.0 +``` + +### Upgrade the other control plane nodes + +For other control plane nodes in the cluster, run the following command: + +``` +kubeadm upgrade node experimental-control-plane +``` + +## Next steps + +### Manually upgrade your CNI provider + +Your Container Network Interface (CNI) provider might have its own upgrade instructions to follow. Check the [addons](/docs/concepts/cluster-administration/addons/) page to find your CNI provider and see whether you need to take additional upgrade steps. + +### Update kubelet and kubectl packages + +Upgrade the kubelet and kubectl by running the following on each node: + +```shell +# use your distro's package manager, e.g. 'apt-get' on Debian-based systems +# for the versions stick to kubeadm's output (see above) +apt-mark unhold kubelet kubectl && \ +apt-get update && \ +apt-get install kubelet= kubectl= && \ +apt-mark hold kubelet kubectl && \ +systemctl restart kubelet +``` + +In this example a _deb_-based system is assumed and `apt-get` is used for installing the upgraded software. On rpm-based systems the command is `yum install =` for all packages. + +Verify that the new version of the kubelet is running: + +```shell +systemctl status kubelet +``` + +Verify that the upgraded node is available again by running the following command from wherever you run `kubectl`: + +```shell +kubectl get nodes +``` + +If the `STATUS` column shows `Ready` for the upgraded host, you can continue. You might need to repeat the command until the node shows `Ready`. + +## If something goes wrong + +If the upgrade fails, see whether one of the following scenarios applies: + +- If `kubeadm upgrade apply` failed to upgrade the cluster, it will try to perform a rollback. If this is the case on the first master, the cluster is probably still intact. + + You can run `kubeadm upgrade apply` again, because it is idempotent and should eventually make sure the actual state is the desired state you are declaring. You can run `kubeadm upgrade apply` to change a running cluster with `x.x.x --> x.x.x` with `--force` to recover from a bad state. + +- If `kubeadm upgrade apply` on one of the secondary masters failed, the cluster is upgraded and working, but the secondary masters are in an undefined state. You need to investigate further and join the secondaries manually. + +{{% /capture %}} diff --git a/content/en/docs/tasks/debug-application-cluster/audit.md b/content/en/docs/tasks/debug-application-cluster/audit.md index 8e0b43e66c..fdceb5c5e7 100644 --- a/content/en/docs/tasks/debug-application-cluster/audit.md +++ b/content/en/docs/tasks/debug-application-cluster/audit.md @@ -96,6 +96,7 @@ Audit backends persist audit events to an external storage. - Log backend, which writes events to a disk - Webhook backend, which sends events to an external API +- Dynamic backend, which configures webhook backends through an AuditSink API object. In both cases, audit events structure is defined by the API in the `audit.k8s.io` API group. The current version of the API is @@ -146,6 +147,8 @@ audit backend using the following kube-apiserver flags: The webhook config file uses the kubeconfig format to specify the remote address of the service and credentials used to connect to it. +In v1.13 webhook backends can be configured [dynamically](#dynamic-backend). + ### Batching Both log and webhook backends support batching. Using webhook as an example, here's the list of @@ -156,6 +159,7 @@ throttling is enabled in `webhook` and disabled in `log`. - `--audit-webhook-mode` defines the buffering strategy. One of the following: - `batch` - buffer events and asynchronously process them in batches. This is the default. - `blocking` - block API server responses on processing each individual event. + - `blocking-strict` - Same as blocking, but when there is a failure during audit logging at RequestReceived stage, the whole request to apiserver will fail. The following flags are used only in the `batch` mode. @@ -199,6 +203,56 @@ available for the log backend: By default truncate is disabled in both `webhook` and `log`, a cluster administrator should set `audit-log-truncate-enabled` or `audit-webhook-truncate-enabled` to enable the feature. +### Dynamic backend + +{{< feature-state for_k8s_version="v1.13" state="alpha" >}} + +In Kubernetes version 1.13, you can configure dynamic audit webhook backends AuditSink API objects. + +To enable dynamic auditing you must set the following apiserver flags: + +- `--audit-dynamic-configuration`: the primary switch. When the feature is at GA, the only required flag. +- `--feature-gates=DynamicAuditing=true`: feature gate at alpha and beta. +- `--runtime-config=auditregistration.k8s.io/v1alpha1=true`: enable API. + +When enabled, an AuditSink object can be provisioned: + +```yaml +apiVersion: auditregistration.k8s.io/v1alpha1 +kind: AuditSink +metadata: + name: mysink +spec: + policy: + level: Metadata + stages: + - ResponseComplete + webhook: + throttle: + qps: 10 + burst: 15 + clientConfig: + url: "https://audit.app" +``` + +For the complete API definition, see [AuditSink](/docs/reference/generated/kubernetes-api/v1.13/#auditsink-v1alpha1-auditregistration-k8s-io). Multiple objects will exist as independent solutions. + +Existing static backends that you configure with runtime flags are not affected by this feature. However, the dynamic backends share the truncate options of the static webhook. If webhook truncate options are set with runtime flags, they are applied to all dynamic backends. + +#### Policy + +The AuditSink policy differs from the legacy audit runtime policy. This is because the API object serves different use cases. The policy will continue to evolve to serve more use cases. + +The `level` field applies the given audit level to all requests. The `stages` field is now a whitelist of stages to record. + +#### Security + +Administrators should be aware that allowing write access to this feature grants read access to all cluster data. Access should be treated as a `cluster-admin` level privilege. + +#### Performance + +Currently, this feature has performance implications for the apiserver in the form of increased cpu and memory usage. This should be nominal for a small number of sinks, and performance impact testing will be done to understand its scope before the API progresses to beta. + ## Multi-cluster setup If you're extending the Kubernetes API with the [aggregation layer][kube-aggregator], you can also diff --git a/content/en/docs/tasks/extend-kubectl/kubectl-plugins.md b/content/en/docs/tasks/extend-kubectl/kubectl-plugins.md index 9932ff8143..387b14c802 100644 --- a/content/en/docs/tasks/extend-kubectl/kubectl-plugins.md +++ b/content/en/docs/tasks/extend-kubectl/kubectl-plugins.md @@ -9,7 +9,7 @@ content_template: templates/task {{% capture overview %}} -{{< feature-state state="alpha" >}} +{{< feature-state state="beta" >}} This guide demonstrates how to install and write extensions for [kubectl](/docs/reference/kubectl/kubectl/). By thinking of core `kubectl` commands as essential building blocks for interacting with a Kubernetes cluster, a cluster administrator can think of plugins as a means of utilizing these building blocks to create more complex behavior. Plugins extend `kubectl` with new sub-commands, allowing for new and custom features not included in the main distribution of `kubectl`. @@ -279,7 +279,7 @@ See the [Sample CLI Plugin](https://github.com/kubernetes/sample-cli-plugin) for * Check the Sample CLI Plugin repository for [a detailed example](https://github.com/kubernetes/sample-cli-plugin) of a plugin written in Go. * In case of any questions, feel free to reach out to the [CLI SIG team](https://github.com/kubernetes/community/tree/master/sig-cli). -* Binary plugins is still an alpha feature, so this is the time to contribute ideas and improvements to the codebase. We're also excited to hear about what you're planning to implement with plugins, so [let us know](https://github.com/kubernetes/community/tree/master/sig-cli)! +* Binary plugins are a beta feature, so this is the time to contribute ideas and improvements to the codebase. We're also excited to hear about what you're planning to implement with plugins, so [let us know](https://github.com/kubernetes/community/tree/master/sig-cli)! {{% /capture %}} diff --git a/content/ko/case-studies/_index.html b/content/ko/case-studies/_index.html index 28bdab44af..d863ae158f 100644 --- a/content/ko/case-studies/_index.html +++ b/content/ko/case-studies/_index.html @@ -1,7 +1,7 @@ --- -title: Case Studies -linkTitle: Case Studies -bigheader: Kubernetes User Case Studies +title: 사례 연구 +linkTitle: 사례 연구 +bigheader: 쿠버네티스 사용자 사례 연구 abstract: A collection of users running Kubernetes in production. layout: basic class: gridPage diff --git a/content/ko/case-studies/adform/adform_featured_logo.png b/content/ko/case-studies/adform/adform_featured_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..7e3be727e39e676c4039d3d7e4fa5fff3de0ebe9 GIT binary patch literal 7830 zcmc(EcUY6n(l=E)(t8bEfzV5&69Nc9dhwAGAOa#u2mt~jy-E`hDJoUzRiroRO*)8D zl-{dUsb74~dCvR&ajy5@cdu)+_r2H5?Ci|kv%lFy85(F)k+YHG;o(u~>VS=L;}ot3 zkP_g24-G%Va04qw%MxRPe2npiq2YMyj!1hrkFFcc32qFBIr_Nw!Ikjv2!<Eisl* zeUJmvO&s=zMjY#g!g1r_DXC&nFb7vShQ}W6@~Imcw8Kn0T!}Q2`EYv?(CxD zi-w!{8kjoxx;iL20#sFal&~P205>=W#)EZxitqqol>vXt1>v6mOalQtf0JNbl>z@! z$`WeGqlrYrd1S@q#2h4LBzWW%#U$3l#l{SNr~Yo#5{Zu7#LOz z;lcNh1Tfsg0qugqxF8Wce~>Ed@m>1!*m5SxpE;R$4|wTH_yHU4#b)hH!xY z!|U=tyc+*2FGv#&hhdOtQzY`~Ki)HRMq-d2&PWuGrsm(?;t__z99$58CPn@j_}}`% z(Jo$aM+h3}#`Cv@Kra793n>XLMFp^&l%l$<%zxj(zjz)0%kCw#B($XEArgw}^3wm6 z7l=~^_{Y8fpS%B=!o|>^$NvN`?&Uw>4@cl)9*qm`%NG4ZJUn(HU9h?-c6!^I3}b4R z-g4S(p_kZCpc@&bM%T_nlMQEl6lhMAo|o@w`H<5^O<@G&_Ta&|_66p_vn(eCr@R>t zU(92$QDbsmh_;$KE$w#>Dgin5&CShMF(3~#`uuo{`69Zg zNKRJvooYbKr%#`9a@rSA?>~HiKYmQjt>V}Fl6PZcBa#1nS)S0>_3B4d*%mP|@m}kV z3QJOQa`oduwhE!w%x%+G7pKUdEpvOP10O}hvLf)jy1Tl>Y+EjO8~2#N3{3SN42+CB zv-RVSQfY&$zkdA^`>RGx*6Z9H4qutBc9lo|h;gN%E)R)(WqjG?>F1{ud3N#h=DIh5 zFGjMBtgYPuPmlZr6-x||lacw{3km2#)T-WG9!^e9%Dtfih96SuE&z8?;u-^Ta&qJb zq)c7}&gJFFEz*Q+q(pL0zs-9!mYNs0sRodq^)DT$QLuqa%FB^1F0^lHMs=MKPj#M! z1;o_R2er4iFD}|gceSeqyX}WmRaHe$vF8cDnA++h90|zcNlnkp{P_WuPgnsdnxfgc zGrASE3-}TXpv$8x^a^y;j^~m!6(pdoK=maGnB?i>6K9)e zlQN4x!d{*oG(S5P6cQ@8tRygR-1<#a;2U)2HLfN(-XHlxO~$Yy$`InrkzibqsJT2t zO32x4df)NBxTI?DtIE)a#5XNhqD{WXo5^=A)D+5yVZm#IZ{%ISmXIDRI&@R+)7j)@ zOF5W*@#Q>d(Nz8`?cx0+P#y^>U7ov)KYrDaK);U0g=CIIPM8um)%}Fv^b%H%WWRf< z6PamwG*9S7FXgIe47ik3`H0jpN-ksXPnbAlp!)jT_uJXvRGn8+F5h)NxyG#XGRvlQ zL*X^~Q7$j28Cl+2lay%`JKfpRXMJF$Jb9lKYLAvfkTLnY(sI#8kOsv`5BY20Z z^^2*_{pJ~=*<=C3bQxnS3z7pSfQc{8f(cS0t_1n4TJja-O#b&ozJ)9W#1y38q<;84 z)o_Z64A?(psUC8yal#yi%$Yxq5$u3}_%cv%UTQ)9JRvO7kI&4|J0BvmdhA=nFH=Sh z@=1)6s0(Cza-seBLNT{c-r6{76ss}uvMVn*%sPBvrKr5z8}zQiTZDJtdH(VI;cqrv zV`rz8Jlgx&I+H6zE0{W*g5l$AwHbc+RZzB-GJJhPKQ&WF_tQFm6!;ANBok9efAm>AXK=99@; zggjE^&0shK9UWbcYI)Hesp^@X_2I(8!d`iMqElqQ2E&K<@835yH8nIeR%)~JID~%v6aLsIe3@ZL zgi!U=GfB_H>FH@{r=drUUZpuX1apj)0mjB-pMwacrKNG-5^sTlKz!EDVplYlb5Y}o zvq~tUHY~7)T^_;lp82337tHO?{|K* zf%|G+zkdBB{T6MmeAYA%{!QdB^`KZb*(cYRzlGs3eZc|GPPT(k2`cjMbm$9%AWG4d z4m~R8>$yS?GDL-iKU?$?k8vxamy67bd8nJvXZs6NwgH^-K6}e6Q4;-cU(p_7{T$s- zw!cR+1LIgq4xB9}&m%4hg*=NYviJ&&MbaqG^i#rXZu@7G9vc?DVCWNkUhR>2d*Js z^@dR}|9H07td}CLPK>4x0%OT_$T6X&+&;^+BMGOItE;OIRIiS3Hk{{9VNbaOl17Rg zEr^)peMy(StDtLEvKRLOVunJ#DQJ)HFP zDDrilHAWE~u;@!mg&Ed~Do|LI+fa=yt>O-VQ7iRaM5;A^+(gE+-THfbO^4^uRYb{~ zq(ZJq1>z`!R0s*jWbz)Saa_K58O3Skq0tql3rKT6+DI6?44R+|X?;O=y#fsQ-Ws6l zIFLf@DeQyy^lUWVU&f(3hSX?YYky;>_UEmeoM^U&{mLG$H;NvD8GZ*{%&q5}ca>(d z^*W{m7Q`ZVdsMI8ArOdsC*WpPzBS64qRhIH={|+nV6{=G)w1{56GFT%3g~50j3q-) zT=~RwgXdjwg;Ku674G^cq1}sD!&v53&$0Kq>n##3xX{sv-^_T^F1ytkML$|PgLz|P zcC-BXvvy26i-lYIeG~bORaSX=wLmw`r;f%An4%!r-ytnayLWCCE2|_wTrpU@s{P?h zF*j*RgW>D*=<>QebH|M06W$6HBA|8qR)PN$kbp8`N6Ud@onE|nF%sN+fb%JBT6n9y z<3lsE<1S{~S&w05YtBod%JRo}-))~Cn_=mk(YZ2pQQgu^MEkCq;=)v1XG`2Sy|GUt zzE!(2f8xoI`PRXFd4NH2W;~8~o)ADhuIhW_jCxfJ`*H7}R(T_qgmRwxl3#}95yw&= zdhKR*bw;D0P?n4-D=Hc@F#E>#hSx|cTh+(?W1>CTn4&r_(t7MG^Hyw-Gm=vYOA|^u zb@a&P0ECWWJzlx{RnBd?^G!G**TmiOrkqeaoepOee>c&jB{qD;cG^%NBIc4E+EHGA zyW19Da=r)X)aY5 z$!$6_=V60a9-}#072Bn|t*7q`555ys<@}mWWqa$S_vnGM#)DXBjE*A^YRtrKyu>&* z4E`xPO2=@%?fm35D9&-zg6e{hSIw}1A~YTwT!qgcQcU8CC$b=^;^IF??~J$}&%6$O z(_M%EquqfNOk=3AszZ}=S6uuS^MT~-4aF3-(k@sc^hC=pG%tvJS#gmu)O&v@n;Mro zX77rEkFog+zuOdYWa3q?u{Jg~SSzC$dLe_)URfh=(}h_Jy!M(;Azu=+Ywme%*AcjR>2&Vl!lTPuXnt==C2~+J|MIWU-B`B_$30aNgYD#Q zbJ_8}qMyDEA`4p;X8?RXIZ#ZeH<0W<8q)(FAR^W{sYz_WsdQ!3>a;pkUnIbRM^hzM zgt}VqqwYnvz7+KHppopCAi~BYp7@1=u~~yFq~h(GZkhf>L$ZvsW!x5yA0|^b?ik&w zs3TTxVWR(Nsr=~1BkH~6k!9l3oEtv?YWP%NE=>zFRpm6y=K3w`6B>>7KOfJ`d;dOE zH9XRB`jd;ZAb127G?J9f6T?6=&cYTA%)7)TCbA%H%3o`1Ye4G;2DgYIj$IgESfNQi zU@?m2OE8yKPj@u8-#&hAQe9bbafmqt-}eWQs5!%TQ-sCb#_*_0SQ9)iT+0xigwGUy zAxyKfF`n;J(o?N`Ki*lMTnh(qRZLlc-6|p^?`W?jYDM0G?b4gOf7^$bR8`P~PQO0w zB+@0(sZHFw{p6_;)UeK7%7wf_UXQQX!o-A-v5UCtIsB-Pl$1tn<6HNY&*npD8Ds)atedcs*r%QISPHMzD4~pXloDg#P^Kz343M~zftUVkxi6`i z?j7EG@H$nod8zMuc0|swlbRz+p>D0s%+$@zO@Yu*mxWYQeB?ypJH*d7SHi13l&rqK zzQAn{Dq#KOdnS#7Yx1j_EU0=1;pI@+<#E5iV88k$&B$2oElE4;P8dZ&vhZEt0b`8pmsduCIJp1A zp4Z|6S4g~z3uigNpSrJEAt1zi(?oCC;jX;{H{Z&ZE@Bd;eJoX zBS_`el>qhW?sc1e#E0@2J2Yg^%o|>33i?M}=-D7~Y9j zDoTlX$n~vQ;LzomEbsAZeHr8C)t6#oi=Qu5LJ%Km=7{l0Y^_29NqH+IYX{|-=Fq{0 zo)%$S^9sZ*59?$rrP3#gf1hI_^k+W}l6|%{qbQ45H(95qfVs6bJ-;~NWmwD=+gbYf z7q!aVh7spPiJx0~>fX|}tVGVe%N22`hRQA4km80kJDBOkv;Fxt>qaj#GqWI{g#>1o zH|7gV+jBO?=g@2cLrzR_niXHeY(MGCPCWtFJtGaLpvCUne$(hE8}7!+-sgyBPi9Nf zweqYiyVKn{KO1o+^Of8}zYKQB_s3O1VTjrbCIH}mk>ui7b)dlvWe&rmA0KSio98>r zrd%-okSG(s0HPaKbjnRsP~6iNS7yMzFMrVto#9wosqQIRUAf9 zabN5d>(r=v8~IUA;(Ny{?4pZ_erOS6%+Xj2zxl$CoqekPjWlD_mWiQdMHl@ze|@}& zrT~N-@ay^MoHs7P#)kF#={w_4<#_&s;lV+uuI|Es&6yG~v$`paXn}m9d!V!IynOwB z5hcZUIc!d{adX`9k#Ts|vySvr+vi_P%e;z>-)^sDlU+lH*FD&??CrCM*JkS69qJe% zU?t~~ECIdOYByRXSOK@ALGNmRQ|x@^J*^ zEYi5SNT{^cytxW3!Z1&~Ce_;gKIG$?1wIE&Pj=nPpZ?o$cP(;|1Y; z=abTv=!XO;u1y|#4_7_3BHwcOpPbPs-Yo-i39_5egE*)iN)>H&*{nT-gzb23QQaw0 z`;~9DUT7P&%imUOZD03RDO)(5nwSVhPHs$<+tzw>*Nlg`E7q`P7qEO7J84j2OrK;p z*XK`@DNP~B{NX1K4I0>crRq0%r6{mrJ8|>1T;!FrGhzJDxmE9&QXw$o3k@H|`1rWu zj>3m_d5>Arwc*UQt!>4=^V2=y zF-@XtTzxjhdUnSyxv-DOvtTMIFCg)9FM|5j2HV@9Uz*jeKYZXmxqJ3SVUL`Baw=fS zfNJq6xBW^1pI@gWCq`N+c_0z ze2>;g4J*5^?UinZ?(K0Xpm_Xx4m>r#PK3%lcn5-iv*o{sSsz&+sA>#q>fEE!uhXL{ zEvz(L9P!&zZ02qpG?=qb6#Zz^vB{mxW?{5r|I|qO(A7+6^Cmaz>B(q=s>CW(D&hX{ z_D%Cixr&$#K%J?kVEG5-26HaFYQj6fhDXpt&awC9r%y8e*V485PD5{fa25>M58N>hWOu@l@+|az0sHuqCqTpnIVs z;@}eJYafo-DV&KLpI>;@;V<-<g7{z!S;aZ^Wm3X&$^aCuCEK8=9%(M)7Eai z9mZI^mHp|cn}_&*$aMI;SM+r2Krc<|wXR5ZZp~R`nlEY|h<)lBGVXn@zw{cgrjjy^ zckJ>yn7&i?2kS+7%eVDmDZO30_c@89eV91~fw{5#fCYdcQ_k-cplu6c4&>lXR}k`Y zrFzOHBxA|Lw<1+Mg+ia)_IubHiI&^SM~2TD0%5n6_SImpg|293pU%O|delNYuB1F* zY9-t3DDAZ9b{6<44RRXdaEP}WwvTi>>HPGQv-u>3=vJa#nq&Xl^t~je8_7PIiS$&l z!mAagHw|;!KjI8y)5B?9Buv6xjf*TwF}LpAQA!#1xxek7VvrUroMGI_^O4`8VU4ro z>Omv2LJPeWgY}eZ$UH*+Ob8b^Ip(&CiVCx%oUPmQKW!ZGh>D8xqZ>Z^&7VKsXASZ1 z!G_(WcSsK_5h_o^K4wkhE*_@JtSOMgEawTD`{AEEU>(_=vX<23ak!faTrDzJ;3k@z zXy`mXX7SZ$K<_Qkl0P;quIH6VSAfkb5JN&vCTr3sXjMWd?k9EEw2#@}`c!K1aVIyZ zo9!UBeVCjtUDqC@_q)ht!^v}N_DPc&`T|2Y(if6;R4-|4FcEvQlOpNu0Li*Kzc83G zh@f~8-Aq5cv_>uDod}E#qh3a`nzgz;DL9> ze7~v~mfyFyq9D25Hng`IGHAiMZ*y{SpTO5DaKag#B)$cfGg6GZlP#h)C+Z@Y1|nFx zZ9P-5w@G0mHUzdFQbU!`c{nVq>c8}$1MS`Ty=<%79YMxXAEEp$Gk7D`wfpuxf4$@i zARgA{=ITDw8BvIj2kvY!GX3gTx;h#!wkQwhBVD4fp7akP*bRJec1Kl}`zT$~0g_+F z9!yz5{^IL5r4beAJ{L9MSJ*C5m4gx`ry%>az}WH;6bc>e?|)JGQBEsOexKRZR36}PuAG4R-OiLg^&%Qtj)O7-W?o_)NklQJMpefX=Q{Zj-Y`6SCzvS4jhn8iN zoVx35(AW-=cVtfM6k0P@ z&86RNeipR#iU#pykqp0)9%$)vIWG})vA+YYeJMjhyROA_!dWpwnmMNpl+pVz=&LJT z=#>OfbtB^mr{>DecmiWJ7ZXw+evn-+H}+GXZds)^+n~a5{&nbjK0$C8!JqzrG}PX@ deh`R{_fV%)x((o6_vfEtT`dD}nTB2P{{xeilN$g4 literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/adform/index.html b/content/ko/case-studies/adform/index.html new file mode 100644 index 0000000000..e9a8acc7a2 --- /dev/null +++ b/content/ko/case-studies/adform/index.html @@ -0,0 +1,118 @@ +--- +title: Adform Case Study +linkTitle: Adform +case_study_styles: true +cid: caseStudies +css: /css/style_case_studies.css +logo: adform_featured_logo.png +draft: false +featured: true +weight: 47 +quote: > + Kubernetes enabled the self-healing and immutable infrastructure. We can do faster releases, so our developers are really happy. They can ship our features faster than before, and that makes our clients happier. +--- + +
+

CASE STUDY:
Improving Performance and Morale with Cloud Native + +

+ +
+ +
+ Company  AdForm     Location  Copenhagen, Denmark     Industry  Adtech +
+ +
+
+
+
+

Challenge

+ Adform’s mission is to provide a secure and transparent full stack of advertising technology to enable digital ads across devices. The company has a large infrastructure: OpenStack-based private clouds running on 1,100 physical servers in 7 data centers around the world, 3 of which were opened in the past year. With the company’s growth, the infrastructure team felt that "our private cloud was not really flexible enough," says IT System Engineer Edgaras Apšega. "The biggest pain point is that our developers need to maintain their virtual machines, so rolling out technology and new software takes time. We were really struggling with our releases, and we didn’t have self-healing infrastructure." + + +
+ +

Solution

+ The team, which had already been using Prometheus for monitoring, embraced Kubernetes and cloud native practices in 2017. "To start our Kubernetes journey, we had to adapt all our software, so we had to choose newer frameworks," says Apšega. "We also adopted the microservices way, so observability is much better because you can inspect the bug or the services separately." + + +
+ +
+ +

Impact

+ "Kubernetes helps our business a lot because our features are coming to market faster," says Apšega. The release process went from several hours to several minutes. Autoscaling has been at least 6 times faster than the semi-manual VM bootstrapping and application deployment required before. The team estimates that the company has experienced cost savings of 4-5x due to less hardware and fewer man hours needed to set up the hardware and virtual machines, metrics, and logging. Utilization of the hardware resources has been reduced as well, with containers notching 2-3 times more efficiency over virtual machines. "The deployments are very easy because developers just push the code and it automatically appears on Kubernetes," says Apšega. Prometheus has also had a positive impact: "It provides high availability for metrics and alerting. We monitor everything starting from hardware to applications. Having all the metrics in Grafana dashboards provides great insight on your systems." + + +
+ +
+
+
+
+"Kubernetes enabled the self-healing and immutable infrastructure. We can do faster releases, so our developers are really happy. They can ship our features faster than before, and that makes our clients happier."

— Edgaras Apšega, IT Systems Engineer, Adform + +
+
+ + +
+
+

Adform made headlines last year when it detected the HyphBot ad fraud network that was costing some businesses hundreds of thousands of dollars a day.

With its mission to provide a secure and transparent full stack of advertising technology to enable an open internet, Adform published a white paper revealing what it did—and others could too—to limit customers’ exposure to the scam.

+In that same spirit, Adform is sharing its cloud native journey. "When you see that everyone shares their best practices, it inspires you to contribute back to the project," says IT Systems Engineer Edgaras Apšega.

+The company has a large infrastructure: OpenStack-based private clouds running on 1,100 physical servers in their own seven data centers around the world, three of which were opened in the past year. With the company’s growth, the infrastructure team felt that "our private cloud was not really flexible enough," says Apšega. "The biggest pain point is that our developers need to maintain their virtual machines, so rolling out technology and new software really takes time. We were really struggling with our releases, and we didn’t have self-healing infrastructure." + + +
+
+
+
+ "The fact that Cloud Native Computing Foundation incubated Kubernetes was a really big point for us because it was vendor neutral. And we can see that a community really gathers around it. Everyone shares their experiences, their knowledge, and the fact that it’s open source, you can contribute."

— Edgaras Apšega, IT Systems Engineer, Adform +
+
+
+
+ +The team, which had already been using Prometheus for monitoring, embraced Kubernetes, microservices, and cloud native practices. "The fact that Cloud Native Computing Foundation incubated Kubernetes was a really big point for us because it was vendor neutral," says Apšega. "And we can see that a community really gathers around it."

+A proof of concept project was started, with a Kubernetes cluster running on bare metal in the data center. When developers saw how quickly containers could be spun up compared to the virtual machine process, "they wanted to ship their containers in production right away, and we were still doing proof of concept," says IT Systems Engineer Andrius Cibulskis. +Of course, a lot of work still had to be done. "First of all, we had to learn Kubernetes, see all of the moving parts, how they glue together," says Apšega. "Second of all, the whole CI/CD part had to be redone, and our DevOps team had to invest more man hours to implement it. And third is that developers had to rewrite the code, and they’re still doing it." +

+The first production cluster was launched in the spring of 2018, and is now up to 20 physical machines dedicated for pods throughout three data centers, with plans for separate clusters in the other four data centers. The user-facing Adform application platform, data distribution platform, and back ends are now all running on Kubernetes. "Many APIs for critical applications are being developed for Kubernetes," says Apšega. "Teams are rewriting their applications to .NET core, because it supports containers, and preparing to move to Kubernetes. And new applications, by default, go in containers." + + +
+
+
+
+"Releases are really nice for them, because they just push their code to Git and that’s it. They don’t have to worry about their virtual machines anymore."

— Andrius Cibulskis, IT Systems Engineer, Adform +
+
+ +
+
+This big push has been driven by the real impact that these new practices have had. "Kubernetes helps our business a lot because our features are coming to market faster," says Apšega. "The deployments are very easy because developers just push the code and it automatically appears on Kubernetes." The release process went from several hours to several minutes. Autoscaling is at least six times faster than the semi-manual VM bootstrapping and application deployment required before.

+The team estimates that the company has experienced cost savings of 4-5x due to less hardware and fewer man hours needed to set up the hardware and virtual machines, metrics, and logging. Utilization of the hardware resources has been reduced as well, with containers notching two to three times more efficiency over virtual machines.

+Prometheus has also had a positive impact: "It provides high availability for metrics and alerting," says Apšega. "We monitor everything starting from hardware to applications. Having all the metrics in Grafana dashboards provides great insight on our systems." + + + +
+ +
+
+ "I think that our company just started our cloud native journey. It seems like a huge road ahead, but we’re really happy that we joined it."

— Edgaras Apšega, IT Systems Engineer, Adform +
+
+ +
+All of these benefits have trickled down to individual team members, whose working lives have been changed for the better. "They used to have to get up at night to re-start some services, and now Kubernetes handles all of that," says Apšega. Adds Cibulskis: "Releases are really nice for them, because they just push their code to Git and that’s it. They don’t have to worry about their virtual machines anymore." Even the security teams have been impacted. "Security teams are always not happy," says Apšega, "and now they’re happy because they can easily inspect the containers." +The company plans to remain in the data centers for now, "mostly because we want to keep all the data, to not share it in any way," says Cibulskis, "and it’s cheaper at our scale." But, Apšega says, the possibility of using a hybrid cloud for computing is intriguing: "One of the projects we’re interested in is the Virtual Kubelet that lets you spin up the working nodes on different clouds to do some computing." +

+Apšega, Cibulskis and their colleagues are keeping tabs on how the cloud native ecosystem develops, and are excited to contribute where they can. "I think that our company just started our cloud native journey," says Apšega. "It seems like a huge road ahead, but we’re really happy that we joined it." + + + +
+ +
diff --git a/content/ko/case-studies/amadeus/amadeus_featured.png b/content/ko/case-studies/amadeus/amadeus_featured.png new file mode 100644 index 0000000000000000000000000000000000000000..d23d7b0163854a5da1da707c3c4aee2f43a594a5 GIT binary patch literal 6970 zcmb_>bySqy+BczeJ%E7Z&`1w3#7H;NT@nM#016BYBMeBFgmeptbV^C5iqZ(ujdVy! z3=JPX$LGA?ALo1jIP2SM-S@rswXfgRYwf+RJ3>cGg_wYr00RSqSWQ(?@9x`s_oU%r z-Mv3~nBBcG9wL=Ykos^K(i4J!V#wLSZJ+=(XNWyi4+^pKa_fakVqjo%IU1NCO*A#Y zws2>D$ZsBgl(Xw^IEJJQ$`xY!9Et?kKXXc2X?HAWcC{R|TkpqpCLo zs_(63VC((dR>F=&Mj9ZA0^bR6h9V&Vl(UnII~XO!^0!>@-TXIMfCcb33-Y-X%fE~= z(bNGbz!6XYh+mY?76=3a#3cBEAaP+K5HCPT5C{?wln?+4@(BXLKtZsOAmGnnxzmQQ zgMsxFmH+6vTS>7vAd#+M0Rc}>Pkv8femKHj04O0LAs{FuASA?h$HC|B<${Eu_*~ps z|B;{wb+<)0x*{FnE`Z+>AvW+ANGX;(OaJA9v#X}&zXiLv{|VGx$OKRjR{QLc1P+#|Hq8~$n9?66a_UD0pGJ8Qs-aN8HoP#2_{ zq7=&=h2PH64h#}j6qJ*Y5V{+JK%kPigpi<|JV;(dOiltQCnhZS4_3{^9SL!Z=$RV^S_9{ zxBn*o-u|2T2h%|I2jjy02lHPF`tOv7KMYj6h07(We6Fs~apfZ4r-@+PLDzZ9=W-dZ zr%wzF@*`;0*D^k6G^e+2k*dsf%ufo_N1IEmqnlfQZX;lin_hb=tsFk?mhzX}cA+CO zC!SG)qBX?~(oSYxiz(YA%no(-Kx&I5b4Z(gq!z-J6~?J&E8e?)>sv2%*A(`OiIf$Q zCL??5C(YlNwV;~FgBnJq_9qF*e=UghtEuN-Op2LyD?zTjWE~7PNQrMX3-6?OdfBu;KJ{H24IoDFL!T+DKcjEF*S?sWc(M^6@IS7w5rZMuosVS1$C=`FUc7 zcB$P3JPirjJbB;~b^UUxx3Zt2f%vbL+JPV33Ez`emOK@cnH+$<+gZ?B46e*{Lg3m8 zCZwsp#g7fr{U{76wzhv~{*+~f5_;GMm`B=veSXTDd6=nkGjVZj{{Fz~nFzs=(8PL- zQ*3#rKXwoeb#E0ABy{=n?z4xg=*OqyY#-y zHg{HIz)@6#4SHkl;E8?qJ*fROyljtF%5IV#EcNnH=~n$oW|*$jh_AExGxm$V1u0{C z15|r+3;-3=QHHL1OQl(lD207V3U#=urjBiQ_ugc>^@5hxovxSUL=XANCQtQPfQi4- za%Dhj7`SZcp(mMp4D$yA(h3!hv;Mr6#ulO>%q4uy zELP3SZ}iOt>uW7P?MoY6GBY$yPYhxrg1%#SpNpIBRmrNbgnYJ>QcKzF^@@tDE8)8Z-6T<_~a>>Lr~TytJQBMk*x!{ZH6*oDH5WP7YrmWH^^FZej$ulY7{xJ_lGA%vS`n3Rb(X8Q?E$JzB$Y?X zt%8S+*36crnER3#o{kRAk!jb_9iq2;C!+>$)63b@XkY8p@V5@)TbqXt* zFw72KXs(}8L&QCn1+qD$^IeSa+A4|S){`9m(%%~{ArWBjQbl2OcM@>b9%mh7(&A07 zvJV-7`S?211}Lbw_!PL$zs1DJ`1tvan3Eu5pZH>yPRVP&qZ$zOOL?Z|x_`KwYAK=h zv?Am>(}~p~XxdcygWO}9ND%t$(pv>zJpfnG31tv*GCipcZ*HGU<1HFZtScHH@LO8! zi_D_uVlL9fg>TZW(V4Z$IBt@+L2QW62!f(5FGD$LcNSZg!k($Xm{h*z(vODKjZ?*x z^nwCIO;tziB-C%hr-W#BnAYa615L>u3p~&w4)H^`aG0R%zz3L^okE^gR&JVxM%DAf z67)EO*N{^Og`>DBVS+2{_g*A(h&RZxln1E}plnM^qV|%WyR(&{NL(?=lF7e+h$JgO z!%2Tri9$x!ucVhpGk(j9yq4pFfM$91rsh)PEM_L(kA2DRjrV;m`k#bzy4k6H{2;IO znSKJ7ilLSY(T3q1f!b}z>dI|&Ol-g7^sP5{cI_+`=G(Iv0;PCQ}F4t*1}QEd01W8RQVvXJ@3WXem9e&freC zISOm~_KWRB@4;M!OU)JF=_m7GAGZpx+Xmy+n26j5A1HX4hzMrUXamYwy)8q{Qq=ig zDs`Ky_O&Kx`o%d+t0)~Mb8o~{tMdlVKdGQfqf_@Cybt1&V|{D9l|mV|hamI}mvFg40oZU|CP2N9!)r*$|fsDMO;ZbhCGG@(TIwhkwcIPhM%^e~b zU{Gx=S=RdzuO$Cpi`45{y0Zj`fgS8Jo{33Vw!1&iirq(NtG4zaTKw2^P2mJHsKS<3 zelYn$_|`efh7snmf@VW(gpyk1dt<>IC?&Agrf6E`9h;OglS}^6F}AMAOw{ZW<}hzv zn$p-I!Pcgb-s>o}gZN_Rq=LKYM=L#$_ue&Mr3SI}%Sns6IH#${rQYDTjq<8;tr!Mt zZ0S9&`zvYGL~5;S^XTqcN5h)XBRC~A2Bc44{yYe8;C@j|^dflFg$w&3XLuS4J$_lew2rbTQoSk~wT4HC*C&H|xa@E2&4P`Q} zUM0J;lPNM6{jtS9gj^SkV#u2zKKIwQeBIIThDUsgBO7){gAGqpJU4f%TT9s48^qXF z%}=%#@z(NZhdjOVd^8S2LejWOSSNpq5NWo&t18-zs@|j?tY>;g4|s3Jjs&5hdWg(D z+0=dNOYaq(dJdjI$fmKeJzt3Tx!%$Ok2>SkE^x%b)s>A!c8w=oyTki%Yj?20O7|+1 zC(_7JBDx%X)$|}*-QSu1?7TwwCAcmBh>psD&%M4cF}dN*i->@vi9to)vT6M!M}|R% zEdf40pJVD*Jqp(2=$M<6vmX{!Pm{J++DN_CSQ&j}$h|fvXvuzHZSsYSz#?@d=#P|^ z9UJcsg-Qm7v#pA~#EO8C6Dsm%XUP@rrnJ{8%?IO?nY=4GZX=O3oJn#Fp9Re<0xPc% zs}kfl^IMEh7lhzTyY2!H8%)A(HY1)>$sCAI(*^WV``p7plE1xn^gPY>c#)3KAhoc% zNiR_PKp*D5KC$)b0A!9vG53GISV87@6Z`kr76`silF!Ypw14LV7?WL9pW@?Rdw`)* z?37qhELusbhx2-9;N~>LnY+Qyy)nDkX=EnO`pGFTUywkVY_^h|i~DLcrnIJ(Pu+L- z$yib{Y0Q0M@JO}@H+!t)x_BLP5r1q{0B>CKij9(btg;xv4qaB$Oz9YHlXOA;%Gj4pIjTWu@{p^Y~06$h!c0Uij zzliE5>@{@mf6|9$nob=1c%BO0M^y;~00OwNI%ZmEgmX|p5)mBqa{uDn%P)qZMxXhL zl_9HS%{@le^ctNHkvMt$0^NA}X}vu|*wS1Q-JoKw>DT2PZQy&^a;O044t_$){&6fP z%A12PfiPOA2(p3QZ|UXw^+gn6#0dcW@eOCZr6U2B7|{U5?X1bIgMx^X<5vwh+(7C? z+D)4Z!sWO-IwnlFEs*Uj;O+`*6Yc57{x065joaPo`tig{fT2K0jY-EbM4{9MD$-=T z!7(w|A9$dIh5$tR#=aTt_X)KOGqGJtKFK@6{aMxxE}Z#5LfDNZpYU0_Cn|fUCSzc_ zxN4Z1kE>p~y>PpRMLk}V^un5r9(xxG6cmgio{j0K?yw&boxU$G!`1ZbMfvEPG>uYz z$_!(e|N2b<&=f~k{Q8+%(sM$4K)qdL`|wx%ft5r;#1eMsRZCG=J;Tmp6e4a|o!C}3 z!RER7&x4@-J&EfcZZK;DtwkWJD3-fAK_w(~j(&9MmGOGokV}%;@k|gd z%W&cF*K+k4Y2ePR*wFc2C5ir5QSR=uO*%f0;F|kdiG>9E&#az08CGUm*)i{kY#Us> zOsH%0^Zlq)5x9fTdV0jOvC-yryGHS{h$5ELA$h(ws(Q}6E&hWLKYub*QiU=a^RhB) zMY6+kOp!%(M6*vlpMut$ zA3_z_8iaAX`W{3P0eaCJ*#0&!SbF;xPdArCoibO(JI3~!@-$bH5sI>1ByLQ=)pF8g zv|@Hg9c=DWYlMSQ|E2Q0n{fjDukO5SyAHiwIKgu4e(7k@o>2IRlCHVV6VDehY!evx z)t?_g)9Vvz-#F~}WF$VTPp0VqdUrBvtKXomeH-hzC(l-f#`bQ$je7@sXc+Y+WxaGU zHios;U7ulmPkv$)N@KV*?C%qW3n6nT3}c>;R4txzZl{l zVCo>M5!8)G2T>c6yLu7EsKLX^N>uDXm zu9bvzosHOu=*M_DTNo>`O*XbH?Jo2 z*g1n;kAjZff#!o33+?U|}2)dw7&Rhi$05OjZz99#?5${tY&Em^-jtyRq>@@&f`- zgEq>K>bkV(v|dA+9Cvz=E+&wC@j%b^dR>aGa9g@!eV=J9I|yN!obZIV@k2YnJ*%-@ z)RWMSPSYrO6jjyxVYTIDcE^VldRC(uGnJ=-=)ht0%G<1NoE}3e-!6<{Qh8}eC>H3_mg1ly~5H=#f8)JX~afSA^lwWcj1KQ&i6UXOH;?TGuahGGOedX>4LAj zKeZhgl~Gc6c-ab?ALiWTF|e-UmC1R!a2g+LsQdqDGph6+UfA8( z%_PC`n0w&hz~~2iNr=DnBsjkDA$e`OFXE6u?R%;{3G%`tlW{@_nV!{OUG>i96_!th z;ZU_1T-AXo=F5r@TUJXYZ@qisRkHoK`O(0Qw9KP9Yi`qR?o#Amfl>=S(dMUR zk5u8~M{y_1k#0(BI?-Ag4j0(f%6ZW;7X5>4@-#0j_I@-MEof(2hZQ}f`Boeil4QF@ zhNj``Wh?i9l$>lo*`UlpU)LF)I#2te<2ZUub)SjPELL^A^2xI&FREU`T9%GSUItJ( z7Trwr=IlP6X%Hih`}DE3N0&0<9!_JtQ`{G|$wXI!2CF2076tk-uQAI2m#>w3nQ{|KO%MSzD5>lF5IITgSevfKuR>jpk1%jL zPU@Rt-yx^QU-XQvCi=-khdRnNFKAwtVOr^1hM=gWrFJgoaqQ~# z{F_gBTR8!ky_IGT@_1FH&2=p79^)(@x!+W!KgF=JqXU@@m#CE{5~~w1mG;qwQj*R@ zX*tDFywQ>fCd~^?e#dQLvMuP8M%b^~OEvYJQ!@W+aUpe`q}cFE#=Q{dZDtl|O~d=_ zqqe$a;rXN9hN|vtbKgl)QqfQ0f&4mC3cT#!;v%?NIPn~^2CL5BDwS=(m^RUE7rM9K zrEgtjRcY~ls4vjz#=PK(Qcfap>V2+F*F7c9VOY%&%>y@nzp{}$xL#snv3qq%YP5K9 z6{xEnhq_-bD+)|@C43UjiHlZ`d~YtJ?`fK|Hm-d7TxD)o0g|1q{ou3wwi^1^E-s2z zVzq+#h9kznFOGt?|4b(wN+-5p+5+*40PR+eu2Y*iNUqr zQG^!Ep(>(zdk{mrKG(mODvjKg209qnzyVx7dwmH%OA}A&3`*`p^BL*u@}k}nb)k)}L6$}Rq4OD0xs zWD9bTg9MJ$xei=m^@9vaUg@Tmjz|u#&AQnf=)~T-`U9v^18*f@*UR$m7V&GPh>c92 zQAPg}idU&AGsOCS`S`KmxfS?P$ZXq^T!(8P#m=EJ^LcR&0|A;R(m6`~y|Ez-6L*C3 zI_a(vEWMZV*Bgv~HIM&4^@;zxhVy@~{H$pC_8tw!MHs%YeirHN@4s1UN?M8)^43BB E2e!!Rvj6}9 literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/amadeus/amadeus_logo.png b/content/ko/case-studies/amadeus/amadeus_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..6191c7f6819f2c325bf843e85d63a35a2f4ff6ac GIT binary patch literal 19562 zcmeI3c|6qJ+s7wFVp3_bmPm^*V+q3y2^nO`zEoz6!7yfq8GH6pDvB&+NwTz9%UZIf zl5AzNGf^b6@3K9kx@+q0zUTKmuixwWYsM_sob$P^>wMqmI%mGVuh%!!Ku;66k#8da z003&i)eWhiZPb_l1}5q|lU4jM^@E)Nw;%!l9NcTKb%3}89soc=5@T#mGS@w>h{8Kd zB5mAtj?oeaeD`zCVPOH&QLA ztF4`)q584!>8Rge!VV-7K@kM<@bHlIkdef@+Ed*j5D-`zBrPpLRY(xMa3rLs1db^3 zEy!PS)X_wgD~3SA;Bi80agjE7Hxf)(crDS7$M@rMCj3Z*BYtN`r3msw5%xJ5%1=TLTk8F4@KlpM^D1o{hXX%BCYlO-pyPwG_V15R`hXifXQCBnj_ojK^b@zg3yRUkHWN)Yhu%u;gK(1G-2Q2De7`AQbew z+s`@w84<0HB%zhpYEl9Wp+1eJ@%wZ_#_vYU{3^1N(thqXdx`v5sl_@PH_uc9DoG|&YZL(;7*uT!@w^r(D1NDpr^y8G}*J=K-9RAD3&qMkz2c`|jNJ2}4uFb$j z7ZI(1fs2*~U7LZ6E+Se10~aj~x;6tBT|~431}<6}bZrJMx`=263|zD{=-LcibP>@C z7`SL@(6t%3=pv#OFmTb*pldU5(M3cnVBn&qLDy#BqKk-Dz`#XIgRafMMHdmRfPssa z23?zhi!LHs0RtB;4Z1c17hOcO0tPNx8gy+2F1mz(p4kt$=}xmIhs$fr~C8 zS^)zWEe*Oh0~cLHv;qb$S{iiif5o-&=S_fU9QDpW59%#`OOp>*sJ90Sq2PwP0D!kB z01yxi04%IhKL-H-H!uJ&ga7~(uLA%(@sZZgj{yMOg<9&W#-1I+30?%j=ZV$j-LNT1 zSoe8}n!>$S2&;J^QSL6?yZsCS@n_KJ!b)XZ1*6UHNtPhQgtiE?;)*(wv zJ?N;+THdb%hz?^3ou`QNQ(J|$urmP=3ClX^*k(wA;vIJnjs53+SIh4A zM)HbscbPwn)co>(gxS>oSx1XpS64je;yijFwlRYplwfJhE4P2;WJJ_(SXw9BwP5z< zE`Hw+o~cT9&9Uc9qK=jJvhdrNk%2Di((H8axnVDzZp6~L{ zn2RVU70f*Eg&zPmRPgegj{*cETE(*$Dx32zEeQ{3dD{%$D_UY7J(7wxF+tWk*9O~- zx-GYscxSJ^*)-3>a$Xc~-aX#aocCh4olfaY+iQ>qY1!K!ujr`i=&-)BGy}dKpzPAv zabxwNjbnrT>!#S^fk!F+4uU)v%qasl0s=jpe1yFUYJpy@Xqjv)W1~}jwO&PrgYij@ zzMj=J#$6#zSNVYE_HrO*wx3%sN9CkNM4-``fgfvPzc4uD{!gvv@#oL^fn# zJs;03Hs)9KY<}|a;jNjB+80+SN17HY4>`Z{o8U2&V_`$R?pXXFzf=EE83w)l*eiN0 zGt3?&+Zld(y43ld_x(p%piO}E0PBj)2+o8``-yiSRtd~au3%EvVyg~-z7I%B4ISbhYu1gPS3>fbVY8hvr=S*#O)|YL4w=#(;@gY)j z4Lv7b9ucz(wjZtBOtCfCIC>5^8qQOFoxk_82qHj+W&LsF#V6rWEWml{AHno}qsvKr z-A%8#7`#g7Mj*>^0&7tS_o%{MK?&b?o-4WDcQ{J&cE+u!v=e4{U>CM_)OLlF1V5JD&1AZ~rw zi%0~a@XMy(dG{-*d+eKTkBYXgtD=DgGGaf*|OK&`d+ju_UL)C zIt1hu!o`poPhC`1Y$y3a_q+2K-!@;$Fgp%{#Q#XGZX*>N|bwQ)*vfM74KtLqS% z@WVilOw57tu!TFhSIo9u2c7ZjMXPud7b8?RY}(_VaH06dhyq9VU|;|u@I(wD#Zi)=aEysN8m)&!oFS-GBl*UeH@a~)?8eu>z5#I>%HtJG!d!Rj!V*Stuw*vQ?4Sjq8FaSqLM zr~K-J92K^{Mqm&0!qcjqu1nplfd;x*_)Y+I(kn~@Lf&~|7i)INn>CzxyUy~E@4g54 zzR1hSC)O!4C-86TONG+~*gE1*gulARR;F}nbo`M- zhi+TqfpDdGtB_-xE~PcYKt+$XzsT3>rI!6w3OUSta+QEm)8ao`Tzk4f`p6YOg@q*Y zPP5}P%F!*ay{&v?7op;L#nSs4KGo!wn{q2}Ei2gNe{SfuP`=WWGoO=JI^_ii!_roh z5qka9MFTf6=1es*5)YV|BkPW80J|I#*^V4OYKz8iDc!r;&tS)!MS@4FO!tm#IKq@5W97)xm6(aZ zK9P-xB5%4qlzLdNA3bI{)^fP~` zb+?oc?k2xdOKeyB79Vf_#_Vhd4{>LRHWSn41PyYN&hgTx3gsb|`nQBp-lpivn7}BX z@aqypO=#AhdH=Gm*=`9zxov=kHu&{ELlyVsr9*MAh}#ZzjGPoZj(tCPD*^W6+NCop zi_<|fD&6caD%K^zMH8)dDS6hCv*aa{yq9keSzP1TIMoLph`dlvY4=^unHBC0K*yD) z=x6}aJB7%QSVy3O6i1L+varKh-HCH^`o292BH;C@-UO%W?%g+~qDX#|U)Xt=41FE! zyk7U2Ah``I?T_3~fU#K(0rd-^#S02iOapgXqu+$aR(6Xao`%3`SHzSwK3=O*J^-w| zRX`$9w2v|u9={`4R2PYJI_6X~mKdYx@gTBXU{|bxG&tQ=H_mB?@xt@lPAfk0;%uzU z1K{;hD-$0-sQ3+j%6uGoUinSFr{x4hAT9~N>&Z%QBK*+&xy~G}9c=qdcD!xfaZ{zh zVX*EZhkcZHqk;XvsT=!lM|`wh9;w9m?Pb zs2gp?`Rk1@cfXnV(zQgY4fcmJ1y|w&q@@LXKlN8OWOYsFe_0&c<4l=t}OEH-AfZtZoBTKiy}q- zJ(c?PNitb1hE*{X<7xhS!3eId#V;vJC5_G7rQ%|oy1V5XkhKLI_wmaJHD8U~*Y?HD zH}_-(-B1YTesKZDe|7G(b1<^Plze9g z=(^>nm{O4$ReW88Vt@GTORgl9i&wcUt4uv|;tO!N{=VA*?|KT2Il(MG2%YMXcY
yom;AHio%kq-(7Ju9>9%#8OT{Pn^w4n}+Rd@&yF5fbzCF8Tw#prG!Qbya!oMyE zATZNikjH`oq(?6#Z@Uf{kbs3|5-E#!!VFj1EdX%-+4C^b(|GYLp{^_$fl>W%98-E4r9+NW?T8bh zY&7TZWSB-i8;Nk(O2#dX%J@ z^wQ2sulP_IyTB)2Zp-yfc0v;CfWc$Rys-VmXAO7PkM=JPLaX|m;hyo^yw5aai={N2 z04eft+;RzPPW?;XTgciB$xPmM7Lml7i+FB{rVJ8`7#A>#?49@p8vve7dVdECJ@CwBL9^1%QU zYcr;_E~UChQ3=e#*@`)w;mlyd?D%IvEhVK^?T(?W2J?L!0-i8#ahOmUut zmHdYlnosJvJGGH1371XVLGh`N`izfu`UU?*{{3Ph zrw+^bdPW>lyJR(4R`1+c)(lKq$Ex63zod2edEk;u%))bC=wLu);~@^uk~8V^6Ng4X zvRn$UoQ|ZXjb7@!GB&up7*-;ObJnyweea}<<5F2ft5ROya{QI5&4A#P`1>0hYHa#D z&RrYxDct@V{OrMK+ru$u>+<}jJk}fi1*9oWef8ew>mvgI1S=<9Kg60>L%GhKjnwe|` z7qoy1rv-;}v9K+wO(9mjjxhsi*XlfgD~^lf^41EE9kjKYV))%Q)jndd5U7Z}n_u;z`kII&r`HZZl@fR6|MQmyns_)6dHkjO^Kd%W8T z?e;OpAKi1SaFStn@Aof-WXPQ~+iGmNd$)GM?EZ+{b)GrRV2>@YZc6VDG2HFrIT!WN zK%*`5;A~?iCz~#XyJ#-q@=`BiEOfMZ&h(-L^>nWTa?pmH8g%8f@zmnT{w-j{IP?A# zem`P`PA7zo3^*UvBM^k1%o>J{I9Ll$C7d0(TvHwUcDsrnIdcE(?AutDvi$qFX#ht9 zu)Mcesg*OUG&Jr#*<`>TbiZ_@y@k{ihyb^xmv+Zf6r*cy^C+A*r7SArE=jfCzpjPl zy&kTSS&{_f+)rlq;AXAty3HRRWC)@Zg9k44r`&-%ndQ&evRJAyu?8dHwa5MLu}awI zUjSsE0xzUja~*kFtIrdsbi00J3;yXNkhbF17+`~@CAS69S6>rr=CmsLE>~VObvmwZ zR4HZO`6hV#XOMGo>VxS7&U%2KOLtelZZup+f$wBhHIh&y2+$1fCC18SS{VZydf*O; z+eM;Ue0*#uADX-!F$X>`?Jm3ZK3QM3+kIg=vvsF-&{Nss){8o|=8nfr4SEI^<-|F* zUC$L5I~XSrwIh1KS~jH%Rz*;(#ecS-cMj%H_u7^f9-5%;Ca#HnMAY0 z{b$}RC$w8kuon(aarl*FEKU?X=9+#{@?jH?BS#%?e`P;eM_IZ2aWWKe%cDE9uuPqZ zw|UieTYBUZ#H9BiGRclpyL1G9^|Iy3Hj}(VTsJSe1b@yyuF$@(IeL9ZxnGTbp&K+W zXJGt@>&q6u@Pp^dsLSSxtcma4d$(n6&p5lOwGxI1*3vgyK3Hp-D`tJ_xb#_1bfj36 zje8dH>OpFuEDz~r&$X(uA*Oc;apMIX`{Yf|Pn8-vZoq!&3NpLGIWo+n+!8yi+Xi#- zX?tj7=mh3gL`-fiKfu25YRRU4SYZ9ilelq;lH|%t?TG*21@~h{TQvDOoA>0UzbqnX z_i7I4d0RUtH}>uJu{N=;!wrVcJrhil`dD>RD(?)UI9tY!Yww8B#B{DGS$mOJjHB+$ z`<{!u&sV|treUr1V +

CASE STUDY:
Another Technical Evolution for a 30-Year-Old Company +

+ +
+ Company  Amadeus IT Group     Location  Madrid, Spain     Industry  Travel Technology +
+
+
+
+
+

Challenge

+ In the past few years, Amadeus, which provides IT solutions to the travel industry around the world, found itself in need of a new platform for the 5,000 services supported by its service-oriented architecture. The 30-year-old company operates its own data center in Germany, and there were growing demands internally and externally for solutions that needed to be geographically dispersed. And more generally, "we had objectives of being even more highly available," says Eric Mountain, Senior Expert, Distributed Systems at Amadeus. Among the company’s goals: to increase automation in managing its infrastructure, optimize the distribution of workloads, use data center resources more efficiently, and adopt new technologies more easily. +
+
+

Solution

+ Mountain has been overseeing the company’s migration to Kubernetes, using OpenShift Container Platform, Red Hat’s enterprise container platform. +

+

Impact

+ One of the first projects the team deployed in Kubernetes was the Amadeus Airline Cloud Availability solution, which helps manage ever-increasing flight-search volume. "It’s now handling in production several thousand transactions per second, and it’s deployed in multiple data centers throughout the world," says Mountain. "It’s not a migration of an existing workload; it’s a whole new workload that we couldn’t have done otherwise. [This platform] gives us access to market opportunities that we didn’t have before." +
+
+
+ +
+
+ "We want multi-data center capabilities, and we want them for our mainstream system as well. We didn’t think that we could achieve them with our existing system. We need new automation, things that Kubernetes and OpenShift bring."

- Eric Mountain, Senior Expert, Distributed Systems at Amadeus IT Group +
+
+
+ +
+

In his two decades at Amadeus, Eric Mountain has been the migrations guy.

+ Back in the day, he worked on the company’s move from Unix to Linux, and now he’s overseeing the journey to cloud native. "Technology just keeps changing, and we embrace it," he says. "We are celebrating our 30 years this year, and we continue evolving and innovating to stay cost-efficient and enhance everyone’s travel experience, without interrupting workflows for the customers who depend on our technology."

+ That was the challenge that Amadeus—which provides IT solutions to the travel industry around the world, from flight searches to hotel bookings to customer feedback—faced in 2014. The technology team realized it was in need of a new platform for the 5,000 services supported by its service-oriented architecture.

+ The tipping point occurred when they began receiving many requests, internally and externally, for solutions that needed to be geographically outside the company’s main data center in Germany. "Some requests were for running our applications on customer premises," Mountain says. "There were also new services we were looking to offer that required response time to the order of a few hundred milliseconds, which we couldn’t achieve with transatlantic traffic. Or at least, not without eating into a considerable portion of the time available to our applications for them to process individual queries."

+ More generally, the company was interested in leveling up on high availability, increasing automation in managing infrastructure, optimizing the distribution of workloads and using data center resources more efficiently. "We have thousands and thousands of servers," says Mountain. "These servers are assigned roles, so even if the setup is highly automated, the machine still has a given role. It’s wasteful on many levels. For instance, an application doesn’t necessarily use the machine very optimally. Virtualization can help a bit, but it’s not a silver bullet. If that machine breaks, you still want to repair it because it has that role and you can’t simply say, ‘Well, I’ll bring in another machine and give it that role.’ It’s not fast. It’s not efficient. So we wanted the next level of automation."

+
+
+ +
+
+ "We hope that if we build on what others have built, what we do might actually be upstream-able. As Kubernetes and OpenShift progress, we see that we are indeed able to remove some of the additional layers we implemented to compensate for gaps we perceived earlier." +
+
+ +
+
+ While mainly a C++ and Java shop, Amadeus also wanted to be able to adopt new technologies more easily. Some of its developers had started using languages like Python and databases like Couchbase, but Mountain wanted still more options, he says, "in order to better adapt our technical solutions to the products we offer, and open up entirely new possibilities to our developers." Working with recent technologies and cool new things would also make it easier to attract new talent. +

+ All of those needs led Mountain and his team on a search for a new platform. "We did a set of studies and proofs of concept over a fairly short period, and we considered many technologies," he says. "In the end, we were left with three choices: build everything on premise, build on top of Kubernetes whatever happens to be missing from our point of view, or go with OpenShift and build whatever remains there." +

+ The team decided against building everything themselves—though they’d done that sort of thing in the past—because "people were already inventing things that looked good," says Mountain. +

+ Ultimately, they went with OpenShift Container Platform, Red Hat’s Kubernetes-based enterprise offering, instead of building on top of Kubernetes because "there was a lot of synergy between what we wanted and the way Red Hat was anticipating going with OpenShift," says Mountain. "They were clearly developing Kubernetes, and developing certain things ahead of time in OpenShift, which were important to us, such as more security." +

+ The hope was that those particular features would eventually be built into Kubernetes, and, in the case of security, Mountain feels that has happened. "We realize that there’s always a certain amount of automation that we will probably have to develop ourselves to compensate for certain gaps," says Mountain. "The less we do that, the better for us. We hope that if we build on what others have built, what we do might actually be upstream-able. As Kubernetes and OpenShift progress, we see that we are indeed able to remove some of the additional layers we implemented to compensate for gaps we perceived earlier." +
+
+ +
+
+ "It’s not a migration of an existing workload; it’s a whole new workload that we couldn’t have done otherwise. [This platform] gives us access to market opportunities that we didn’t have before." +
+
+ +
+
+ The first project the team tackled was one that they knew had to run outside the data center in Germany. Because of the project’s needs, "We couldn’t rely only on the built-in Kubernetes service discovery; we had to layer on top of that an extra service discovery level that allows us to load balance at the operation level within our system," says Mountain. They also built a stream dedicated to monitoring, which at the time wasn’t offered in the Kubernetes or OpenShift ecosystem. Now that Prometheus and other products are available, Mountain says the company will likely re-evaluate their monitoring system: "We obviously always like to leverage what Kubernetes and OpenShift can offer." +

+ The second project ended up going into production first: the Amadeus Airline Cloud Availability solution, which helps manage ever-increasing flight-search volume and was deployed in public cloud. Launched in early 2016, it is "now handling in production several thousand transactions per second, and it’s deployed in multiple data centers throughout the world," says Mountain. "It’s not a migration of an existing workload; it’s a whole new workload that we couldn’t have done otherwise. [This platform] gives us access to market opportunities that we didn’t have before." +

+ Having been through this kind of technical evolution more than once, Mountain has advice on how to handle the cultural changes. "That’s one aspect that we can tackle progressively," he says. "We have to go on supplying our customers with new features on our pre-existing products, and we have to keep existing products working. So we can’t simply do absolutely everything from one day to the next. And we mustn’t sell it that way." +

+ The first order of business, then, is to pick one or two applications to demonstrate that the technology works. Rather than choosing a high-impact, high-risk project, Mountain’s team selected a smaller application that was representative of all the company’s other applications in its complexity: "We just made sure we picked something that’s complex enough, and we showed that it can be done." +
+
+ +
+
+ "The bottom line is we want these multi-data center capabilities, and we want them as well for our mainstream system," he says. "And we don’t think that we can implement them with our previous system. We need the new automation, homogeneity, and scale that Kubernetes and OpenShift bring." +
+
+ +
+
+ Next comes convincing people. "On the operations side and on the R&D side, there will be people who say quite rightly, ‘There is a system, and it works, so why change?’" Mountain says. "The only thing that really convinces people is showing them the value." For Amadeus, people realized that the Airline Cloud Availability product could not have been made available on the public cloud with the company’s existing system. The question then became, he says, "Do we go into a full-blown migration? Is that something that is justified?" +

+ "The bottom line is we want these multi-data center capabilities, and we want them as well for our mainstream system," he says. "And we don’t think that we can implement them with our previous system. We need the new automation, homogeneity, and scale that Kubernetes and OpenShift bring." +

+ So how do you get everyone on board? "Make sure you have good links between your R&D and your operations," he says. "Also make sure you’re going to talk early on to the investors and stakeholders. Figure out what it is that they will be expecting from you, that will convince them or not, that this is the right way for your company." +

+ His other advice is simply to make the technology available for people to try it. "Kubernetes and OpenShift Origin are open source software, so there’s no complicated license key for the evaluation period and you’re not limited to 30 days," he points out. "Just go and get it running." Along with that, he adds, "You’ve got to be prepared to rethink how you do things. Of course making your applications as cloud native as possible is how you’ll reap the most benefits: 12 factors, CI/CD, which is continuous integration, continuous delivery, but also continuous deployment." +

+ And while they explore that aspect of the technology, Mountain and his team will likely be practicing what he preaches to others taking the cloud native journey. "See what happens when you break it, because it’s important to understand the limits of the system," he says. Or rather, he notes, the advantages of it. "Breaking things on Kube is actually one of the nice things about it—it recovers. It’s the only real way that you’ll see that you might be able to do things." +
+
diff --git a/content/ko/case-studies/ancestry/ancestry_featured.png b/content/ko/case-studies/ancestry/ancestry_featured.png new file mode 100644 index 0000000000000000000000000000000000000000..6d63daae32139867e33f6fe0d3eb68a36f0c4ded GIT binary patch literal 21016 zcmeI42{hE-+rYot2uW$tVoed51!I`(TbAtm*cxNTI)<6ChNL7Zib_b75VDjkS^Fgw ziY$>`2}#z-&imEgRP+AcbN=r+?|IM6Ib-Iz&wZYI@8>?xy?5?8U!i(Bhc+Z zyvjrpju!!u6ve{DrFfCj5I90wN=)JauNVw2357{R;S!=SI0`0>g28#ezWC*s!EZ7o zJ3PujMeS>Ipd`!hK%uyypinO_FNl{ogh;Z7!lk99p)fJ1n3yP7LzL{}O2K%Gx{~*N zaq`WN3XY5=Il56CiLSgWelfO04~i^5|4KvOUth;XaQohnEBPxskRsF@;|7I8V9>u9 zX=~FqBM|;*Nv0U!evIso4U&y~+;C6>9GU1r!s5W{d;Zh~GDX!B9MV@r|H6@~C(f1f zM^|>(?|$4oNG_j2+hL(N7aRdJA%hg)e;kXO1Cc@`I}rZ@@Vol27%7hUzpz-*{M8KS z?f4h770nm3&oT97s%5}HMk$kU7z&YOL?pV%eF;~+Z@uSLR$d8DB(IP*2J7g$GNk=7 zP->$eq4UEZP6b23$*tIm!jPh3FeA7aN?ZaZE)I$)80;q}UushM&?eeB;(dPeA&!Dc zp|Xg~cL~wziTO zQ895OTvP&Mixb6&BjBPq1Oji1k+hS>Ny0x5@>lcUT&obV9xI{r)wLbyT17%!ToS2* zl$20Yfx}giiYiJ7DM?9HRcUcCaV7Y7w?CQw?iNRK^u*bzk%)xPS?-1*k#Q@@Bg_9S zd%iCVUqj(@`k@?uAF7&^Bm$05RRa5?{OA7sX8NN)pmUioYP;V@`qi`Yf9Lrp(;uF{ z(N*~|RI11Z(d&{N!G#^;@)N6-8T_AM{)3C3IstkEn^AWpgBj%WQ~N99URhoZaqhpC zep9$O{>zE^K0V-~hQeT17H&B*2Ao_O=;wv*TiK6JeX&?+=u3q^%qb3zhv9J25~5<# zIGCse7_XujJEQ~{zP5P0Bp!yrAd%nv|78C6HoP2guD_%^n5rwA*|${v(v#1%{_ccK z#8bR5B%Go>m=XVOE&Ag3SHr(IfOWvQ+T-lxpx?&!t$^A@2D&nNE{^{f=kDJvJ-r=$Diiz=VkW)#oT?F(O=W?+Y<77`pVd0QFtPWfT74a5-|2SsGF<34D?6g512CF z_i~gn(S=9?_amH~xD519#XniC?BXk{6By{2&nXQq>%Wyz*Ed^PyInj!*GG!Mz!wz0 z^7pIh&uVH@bK0gqtEo+YSQ@)JQsl(GwnkO_lhw*m;|mN!@OX`rlSClI5i(HfVrmOL zM{k^qnTjL09glC)rea2M@#b`mT&~v$8)IxgCLsI+ukR6uBpbX>IgP-*G7sDRKG z>9}a~q0-WEQ30VX(s9w|L#3tTq5?u&q~oH^he}JwMFoVmNXJE+50#dViwX#Bk&cTt zA1W;!7ZniNA{`fPK2%ydE-E0jMLI6pe5kZ^TvR}4i*#JH`A});xTt{87U{TX^P$qx zaZv%GEz)t(=0l~WA0wX&=%>qX!D`c(s5A%p)JzH z#r)&3S)42QtgILKP^?y@I3M^xEiYExKpOyj_X0rR2>_T~20sS@zyk&V?~ecg>IwjC zCq~*-ssX?@F%1<(Bk#5k9bO6?UdxPQbvWg97~NJw1yzNMd_C#&v*v^Z;XIR}0H-_; zBd3??ndn2Ed2_dn(^h+Beo9NyG-FfHS5RQkXFABL{Iqs)>AaRQYxuxm;_$%KcCYAS zzo_DAugy9a zf2p!!MqT-T+h6%wHB;fUM*2RB^H-4%MeoaP^w}+#_M*cOPu!HA$1yo}bJH_dXq(NU zbdkvHjdvT~ZRt=?!0os8-1BB^OH#R!x*3H#WDQrTzR8D9A(2y0A~S%kM;U;?n*408 zHmq*2bPeQo?&A8yTnW_Lfx*q8_S2@MMKu15o`c2=rW8v3wOVfJV@~C zzIdZ+WHhzI5r5A^!^NdPO%tfdP1s7{}r5vplM)rPYv6=-l1XysrYT zVXDARVT80_k45*xg3;?bpUJ=Qv0V<;MO-2)MKeDl}#*LIxvGor)9C zsAi2|1Y!qD5{&pS*dY*-_`LE@xSIWYH4L0`G_eb#7x0C2LNavCX#P$^ZexOQJE%X(3|Z(*Z->B7S5bRU^( zMR#umAqkHUoYNCxed!>v)uO;9@~xfy)G9y(2z-$<*5Sib>vgl<;Zyvj>p5fA@1vc| zm`pQXc3sE3Zisy^EON)}*|{(tx5my2<$#e7<_XU&3RWw!o2XqcD#tad`M8b@U%Pzy za#m($=Bhk2p})7+LT0*RHWMtnp3;?_k}}u!zRuKF=#sFfbB57VxVW>`J%{nknrLW@ zI43LV@w3*~05DW$NuJgj8D@*(o-gn09eh1_sdHZx+}LTWa+;o)$?ds^8mWyMrUz`= z-oJa>^k%T{WwX!)n2QSmae}dYn^rJkAe`V105zt!H9aVl%?B1G>T8>u^Q+q1Q_Yj= z*-`OqXP8-rOZ@$I?$eFgxej1WGm0&Csua@_agGy|lamu|H8&6MwFyZWlFi+jqT=yvt#}=Pl{dqjkU$^%u29N5i_ff)s+X#`W;`0tNEU z9#gY~9D&aqZ^~!!X$*URtz(>rv(Wghz*(3%M}A(_L`o@Ax-fTh$#}nf;}Y@NE6xfJ zr@W#MA-UaQy4R#qY|60%SuWfW#Dk7+)*ua^Y+??cYA^4qtGWw$JE^+U@7+`QHsFfkJRXm~`QcUmUN+!J{gcoe*GiRa z859ceQ|U)%eS8!Fpwa%BX6U)FQMU^T(*9z@Gk4xzDV$+If?(d?)X%U$650}zKkbn4 zemYfUnDXh8X7YWW=W)G#eZz*dm4TPUF0BcsUgy!XWB%w3$=|;jSn#p#=&N4&A z)JCg`k*-btrP!k{q69^ohqDPC_lLSoq9Io7!`%f171A^K=`PD4-qRZg%0^q8n;)k( zhfIeSPAy%%pU=83Y1hlPy!`yjBd^LXUGtjO$tgXd+;_K6#5q_AbITG^?zQNPkB^U+ z6@|fWEq7-YuLah49huIooQt{CkT+6SYg>PNZr=X_8{n4fQd)x5dT1=S7@4;(BFiL0 zD;0zG5;w zEXO}D%oiBPk`_*$Y`Up@C}Vo$pq@iv>HYhypWaj^jO^U1nCo31)cpSadsjfgwd07aVP=EsTwQ@7zT4 zFgETV$5z3Y7#OBgv=esxa|Bl8{_NSR&GP0uCHx=X-U2RSI)?yYY<9tVxvMy#T*_S| zClKn;e5`p|{^_#c6gmOgLZo!x!^M3Jk1r;?n@kn;bBnlk2TX>NY~paIWyYB>X2Xqp zxjt}oB-wK-vZKJwG(Jlbi7ePsWu@ElJtKWz94`1E@ZbfeA{)+m)*j7gZ$I#KC9)YFoE6ZBWJbPxnDgpG z{@9qD>{+D`eed6!sEHmcCPtK9F7qB2+zBWc8b&?vKFhIdJCEQN<{=So`E5J5Do+a! zT&&yfwqe}@*-t9fz$%#g9tI~I(B_)% zXw8*dKl=$Q`kL#*`&At7tamol5iA+@6-+g~k!+Ip_2JN12|> z?lg)u(HZHrf`j|vDM6e{PNWz=+u^fC7gzyHK)qDINOI)bj9z;E;VV%Iy}McffccSz zZa|FMHiv8Z0x5Ol?Hq!$5xb`%)D7F;FtRq;bQl}kM)e|4z0r4crXM_5r6rQ#)?0O= z?J zwLl>DvHlLrTZ&Cb%##f^0YDxC0b%K{%ZRM9<3bleTaLYi^-tYFJbvQraI2+kGFT;P zM}W;kMP=)Z*KSfv004n>-e42p!nYojnnZL6^1Y77pU_MTQG!5W+(!`-O$W7oz0RbO z|Cud)+*Do@_w-_(xqL@}h(abIY%SoH!z6lo<12Oh7ta?M5>o>tGnQ;M+ZL1E0oeZi zdu%rY2Vyxr_DJsE!w*mO8+v~*!mD5wRdP+_t#3x5%TctJMGOK^aG$E`;qL4bx!(Ay zOTZ!Y)5tbQ-`E)Ml*b0bmnum^SGOloBJ^-Cbv6#Y;y8_P4Db7QQO6x?ZbcUEi*6#hEX$v!Y)7vfeM~2O|PiCx5|l- z(KD8})T9v#052lb2QF)6nnd$x$r()b&XO+xJ=?d-_o!)UiAS`#;$4>>j;xDUJC>}m z(b3sCM6doGWlk%9GPbA)?d+5s5iy!{pQm;}c*_|L67b;p#SGIqy(jEpm9ARR0!466*1k3}fu~i)*e+G)+UeM-Y-xVr^t@KLp!?kq&K=1)sCZem%;x;9u~~zI zIu%DpkR_$_Yk&rZo=+dULcn7wc*GO!fr;*FPd$(FCwO|AR2Pp56fS!&S=u?6%?yvS z2_m)Qk8 ztsfd^*s)QrCUU=XuDXH6T7_zL1Jib);)L^>HyVr7&O(dv;SKxbkZy%?rdv6UMpXuTC;RAP~|Fu{QXag2IP_`KZEOlCHHB zuW2E8N4|SSk$P`jqNHSgxsUD%wWQW^IZw-Nd46vm!JUD?QMQqRwQCs%IM!gMqc^Sx zfOAKR&W*%>kSp>crEG1$q*y!S$Uf)< zw(8K-^zKs|c3uCVmn-jcV|3|C9NEV2(7;xU&D+BIhWq+56Xq6P&xIX2KaRGsFl;=BCf>CaXw^fbBh^GlJ9FpNz%1l)YOSx@KM{to@g;T@^{$Jm zy0{u}$V1}|vDo1HyzAL(ns!bRJ>?1SayHHF%9>_oUR6`A*_e@@?vkn#*SeRRbKWX7 zQ1n^HopTbEq&%Z4;Qyn-}bnH9|vf%fr{Xz^Yi}>`@ zCy`q=0lQN-9qv0B;E3}KvgUH#ZH%z-0)U*?Yr6!QV)p=nH)I8hbkLhMluraGv^F$! zU_PaQ=O@COAm;|BJmYnp9R9E^%xgO*w-hvcccem}U7SuCv^bfPvadL|JE3;)=2awz zyfr7Twr>Ie_>*q<4^oPjq!O~xuU#e`Lc1oCwFFXiv78YrD$x=g+0f{d9}ic+eS*jyBgP#<-;1a>`6| zC>pl+;!CZPGBXK}x*-DPtls7$`3pT)MszrAllD!|yYSzF)J^K!r`!M%OV(}O+Z-imm&qNSzK!n+1McdO4ZN0I0A5HC-4Zn$vD4mmXWR6j}lR^QvCL$>+E_Q*^l;V_Lz7yUG&X)l-V*u{UYw)$TdwYlJg$2W5U zfjX%NGcEJx*5%n)S!I6w5YuSfw~;4hh*E@(6xda1>yCdogH*KK>uH@chFvtuO-MLq z9$;<#qBJPT{xO#eTgV2Wr?1b-MJB8XDrRtaF7fXuk zh%6yEnajyP^rqsRaLc2i-mW7%ArU*ovlxm_LuSmF11(>kC-Xb22muN_b;11dPu%Nd ztS1sboqgw96>A0ru3v1ZzQrD2qb)>;yyP7W##mdY^!4&k^US+xhWqY2NW`+`^(X$L zoTaON#X%a|wgs7o&P<5KiN%~*WZdX}4^8MxO*mj88?bOo+04qSv-&b*`7qwKcEdi6 zyY9;cM@MbSD0eCrfo*Y}pGJxe@7nghnM&wQUB0?~FPF@V!PO@-%FLoh1_!qiN~{lT zvxlI^Fsb7DSy`^h87*04UM8h(8ySxY@3|_DCrJ!|l5a>?3}W6tsN%I*Xyv7d%`3s_ z>bhjtQa)XEEo=xIjH(!e#_?rjJ*MpXP(c}&CAFsf% zxrfdzg)G;EiKuIo(uQ12mz%REs(#|F|co~WV%aQuX=Z$n#UaKgk9~MVd*CsLv-|t|uTDx_xb5Y@8Ksj1{$n{5xMj?H zhNlqA)yX0vL$FK}3!lL^y2)h$OB6Yq2Te{06e?aZW3#C+ML^@!$;oiZtDfLbah$wO znVI|ah?N$Y=z-7i8K>BI^j~J%*a<1JFASWOzB%jTMGRHh-%-?tK5md^cmJq&lVHgr zn=&u^$t$l^TV4s>_n4}#J0l`*^NEA$z|C3+M0>;3RLSc-YR!|O?{>|XOlxjUertzM zX@|ue7=CfRBJM=c3G6LvCXL93j{~(`{U6ufEDm8*&NOk3IlQm5C|%PEjea;cV8L-! zTh{nU^$Eu5MmdX5j?cz6?Gfkb3J~0E5vgM&nUZWNYRL|6I&a1H?|DSv?Bl*v7f{$5HrL*QIZ)%ho5V+42i`~RewgTPm!uq?{!h<(^Udlz05iGFK7{wn z#j4X_2%S{oOBGprvnE(2lkhxRSop-PzJ?+)GefF{rlH09kC;VWYjAdXfWY`f`}*@e zL>H#S`@1|U-U3Hjc1Dx_K+ZqsDM}6K18~c-kKSG(-4v#0mLlss zyZQUwug=A`*`ziKv7)!KRwbN9XF^0pM5boI-)hnGvue@Ip_(60+QDZ;{ldvy@sFEH zS4;@aC4nZ4VX&MxvqkkxDZ2p5Ty`)gMLa^EFGPHAX0Wh*jSz#iuT_+0a*>VoQqJR+ z7UH-ATCm*Ks9&4UZv_`ZRR#*&f}Y?1BpJI!k4$d56(_yKbZijrF10dzW^=FWhVdt literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/ancestry/ancestry_logo.png b/content/ko/case-studies/ancestry/ancestry_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..5fbade8decbc19b9e913fb9a7f0cd70d9f152ab3 GIT binary patch literal 21141 zcmeI3c|26@-^UN88lol?S;rQYeK+>qRCbXijF}$%*^Ko6@<(&7moa>zPI>A~RsuW~&WB>qAz}1v=@V{^I ze|}(M{P$O_X9)ZcsiT@9766V??fnq}Nh!1dAkJ&8cg6XN`Xv|=W6z5~VJy(R?)Hv& zYXFdvb$3J{ZPCt93$&HBgEZ@6O#>^`8YRtYAfnE%?x={qX07Jwgx2-c&_jCKA|+6) zvNB{+?l61+d$cnG>TYl6fQ7kBvwn{Y!|T0bK33>=6K7j#R{6bz&@1X%P(_Rr8Y;po z%7YXT5`>CL@Ct}X2#O1HLk0N-MEUrI`2-|*1cYGtk1#*<=f)~ShF4NfC`*`*lFH9? z`2VC?uQ@wA!ua^y+}wEGgm^JdR(N*_2|j*7K0!eqyaf-|!@(Kh&f|b(`w`^#I7(j-HVH`z_>U|v$F0b`nCN$E_=sci5##$+2JYjxg#9;1bF%R{>cf2{N>~5 z;$-)I*HB14v>nKau{UCsxnH5zVKA#$sHYkZ5IB{GqV@)6qLyTmI2Fe~PqM^RN4kcDMc)x4oJl zZr{7Yj~0@`A1h4J35{^ZIO$<9b}~PDnbz+Jp^A!oy-I{v1j?n3Kw3NOkv%KL_gA++ z+We1*XeER*T4t|L^6*RGKY9Y9FhL2JFkXwn`1uKf{IL8>40R03+S216Vh9VugvAMC z5Lo^-27U~o5YCAIRTM&7f;cE7%o5{dk8qZ;wntc@`5Yarr1<_QB(VI|vtWuCJB$;4 zx6m>|QhfhZOyKpsx53~JSZ9O-5)D_9!58paTccn?0%%K15lcZHq@|bykFdCaIFC4j zUy#RA2#w@rX;}h8X{^?3s{a07R|LjUw{a4pNJTE&~ zJIe_E%=VXJ0$QzL;5cV zCJc9wgpkI*`T*CyhzJc1a1qkjS0CWo7ZIVs0WLxs`|1N+`ywJVIKV|nV_$uMYhOf! z1_!tZY3!>HaP5nT(BJ?UA&q_Y0j_-!5gHueBBZgeKESmvB0_@$T!b|C)d#rtMMP+D zfQyjEzWM;yzK9474sa3D*jFFm+7}U_!2vEp8vE)4T>By-G&sOTNMm1pfNNhwga!w= z2x;uA4{+^^h|u5w7a@&(^#QJZ5fK_3;3A~4uRg%FFCs#N16+hO_SFZt_C-W!aDa=D z#=iOh*S?4d4GwS-(%4rY;Mx}vp}_$zLK^$(16=zeA~ZO_MMz^`{U33W{qYhY+5!LO zpBw%aKh@bX4F2^%C{j&F9RNHz0Koqi0BnB4|IPq_3qJsSHU|J$0szopqRd)U0O0f} zTuENf{ry~uhqqo|-LtOh*`Wc!L=mb|by0AD+&lEq?K8wWX(vz6P?vZ0AJTqfQ&_Pp zHLJL7HJCc6_H^KQreC^Cx^W0Gohl?N|=q;(2aXZdPUR6MD z;nkx++C7Cg#H;_Z3$EH9VsTX9TNg9{AAE=n`715%Vo>Ne9j#{U>6&dFq*V!&I7q)> z0A;r#uxb@mm-XI;z7EBCQ;2R9VR`G>7{8>#`#SSHK8U#@KgJJ^u zKgUG(AQuD*tOTFT1z82Q{5H|qRG46+9U}?4EFn{k=&bW-G)`GtVuB zL1yBNl0$}z+S%D=#$X79NC_h6&oS4!aMf&K2X%xLo|Jn-e8^yNe!hIr-l^Io{~j?I zjxJ>m3WO-!veAjQ7Xyd`9yDKvbySI#wxh0a=g7jGaJa*>Q{C!`y1Hgn{aRGQ&0G;+ zE``pBk);Qdai{L)utC7nt3gjzoI|#@nTm^x`)tb3#lHIVi8^X0tiQ!#=9-P2Q#FHA zbrUUpa#R!b56$-SVsM<5#hV< z9wR|S#>PgId{-ibtUerXRsf!XJBaf)harnB_cI5zPmKAHbA-+bKFE7bw>sh>LSOiL zk(SyJ9>ihbh0bns~g8>Cg8L6ny5NWq zw3CFa`?uZN+AN*?7pRVu7h?)#cPjSaX!o0ZFP0h(ysJy90@h6na^t9=wq#>2XhN|htr&b z^bR$N-m;>nx<%8~ zcjfj-!{E>mcgRhy5O(D#_5pNxTzq_9PN*;~c_f=*Q1=Y^nyD8;zP`mmAlXpw`*Co%UV&2RhrL2$;tCp zuTzw@^{nC7?d?N*M^l77mZ!UsC7_MbFJZ?k?FVshgSNJ=kC&wnKk?{$Rnv;HOqKM? z&8W|K@`P3)VPH|(+Qw#UmA=95+JljecT~2E#4_SyssC+u^(ORACYU7v5F~H^s=&oS7N~H;-0{tWdoeN8mt$VdyV87`s;sOul$1

F@IXk7{R~IAl{6Vfth1*Ju5rZ-rFNgG6J<%!F0GaJMjF~sC zG9^V~mf8K)33Nae$hYN!5{O% z+;Ig~kF)8zdFZG?k%@%o%C+0QqfW5XaYt_}tgfz>S$0i|FI>olLvYLSvOZY;c=nfP zb@LoP)GD83RFBHObcSBcq1NQg!-66JKx-yM2zZhMa?}CGWp_Rn)_Fk`&P30&#G4Ax zA+3|bf`cQ~*wGU?IvV;u^%n|zIXLqGz&*`x-3?=)um9=OYr{$$Q5+gAXn4WK#?#2N zrMm2focXJm{QP|D$B)?-j2_#}luk@cpoc4XLo$`*-FHJrig4XY%6Cuk8P%fNdeFha zaEJna;7B7Zbdk4OEF#s!<1)%hUo@JZ??arPFC6{yAgsdsJ+xA z2%-EuJ)JJI`s5r8w)~O2d3wb&)W~OUx?A~foYYb@XI54pNd7lrb!}?-h;MF(8sceB zvwT`EFL$LevHMn9vD&*8RyV(95#FGy^zuQfl;rqvU>OKeA2u_y+co}1bva}O)l3I& zYzgD@mLC-HA^NDCe&;3$ub!D-^|TW1B46F|&{yMy1Q|9C4wunP4!+%;zJTm(N3NIf z>fE3g_?|xdf?4=96;=CJM3QAuYHDEMrm?B_W?Q;Ux#Q>juO9A_RAKiynz+y=C}-`^Dwu8Ts5h-6aFrb-ib}KenCAkwxGWtPfwmz=pmaAJT zr!y&3j?xK43IM7$A|a0-C#9^s^NL5!pDXa{TqrsBbmg82C4FA|%yxkGi`(=tH%afl z(0+^{RX+*9Do@Ji-gGlQqY4*)HzdTbc3^ zRceRy+4ZYmovd5`Aa;(8&1q-Ij87%osI*RgI^r7-d!V%~V$UTl{T zxe#`8E?Y@AVM_f>Zy#6071ycdk#7r(U<4O6w^XF8$8rn~*^l%X(MXdL5fE6asNknv z?sxV!8CIx&P!3E~_#zWz6k44fgECTF^2eJ`Dh4#v;IQ0S;5EJfo8^D(vnV)=%Serd{kAp&%0 zr^K?nc*HOuh9RR#Jsz))vQ3#K7X7c5w}!EhQzq>0L)yl+-@4_asmFmzHsww2eDESD9#(tMHIz zLcvxV=|Q0D6GmVNNWk-wWp)nf;hRQv$AJ6Y*Csns*V{NHNkbwoNZg<~s!2^vb22A8 zo4XFXS?h6hYrQPe?ZKi8osHK$;s(ms8Y<)8Lk&{ zr!*4rd1LIvE5q(g!bKV3a>VG^# z*dU)Gk3*p>C%;@!Wj!}g#T6WUh67CO?_knjUA_Im^Q%~F>`71LYuksrpC=4_J*HCI zDO3~_j?6J0krb|WexNcvcScPEL>cgQHdsvploF$;1_H~efp%_~ zL&e2!kEk|KOJ84n@4{EN^Lefsdpc;zT?&zAWvHK_MSUvmU5#58P*<`;ks1#G3S6xG zRej8t6fAl+mIizq!VwAj*!JqOJR)GK%V5B6Sje*TX|dH8sQ;N=oxo2Yt8+(^yp9EW zNGhN6k#BF?nxZdGZ?T=UiKXI*F@;F!ug|E=mS`B{2MAwM_d_}l?YbJ+d+g4B&`OTF z$e59MDl4H=K%mw3?Y1Is+#BA`_lz83sn$CW%Q_l@qUfpJRyPYYT^9iST5I=9b9h)A z1N}hf?WV|TSuhJxaO2^X@;i$6a!csB>Bx%$R=0Pl;YP*!7gcYcmGHPD*lpv&-Od_t zEu_?lCp9&duGn?j;?pL-!kyg2$A*T;veNhOjgFEnYsSSrsjMiaj6)AZbF^HH=MG(< zv0ab-25Dzd_AjZ|s1zo8>exNjlt~2+24}FT9-I2~y1iX84!IiIHfm3$Oq2m><;qTp%oPWL{5M`2OMlF4X=yPHzm4WNtawS~#WXG$4hDlD zwWF{%FU3jkvm_mH-sQ~_5bdY+S{hRhLYQb+!Qo33JahB&JiM^S?rl9M*v6tRwN1uZ z1}m|KOuK*R7D!AKtJT6S)y~b4YM<*H5VAVP^sZrZlgBBL7=X~wG&l@zNiNpT_NFp3 zOS%TI!z{ni9 zT|S=f;0sr38Y_C3mt!By1qK}5XF=2f5n4{C8M250=(FaR=H6Nou+*@Z)NzCJ6@&J> z#B2U{K*W^bjF-=2o_$@bkD;j5xek`I&>E#=zwxXuhsuCrQruyocC*nOP*Br;BX$IP z{`|SaDbubsxY^WV7ngE&c-zGCp`$b*nP5nSV^mWURqNNVW5=Rgy{qP*;e3Rfl}h_L zayanUeaw(K*{yfuGc#G?0op(fw*=AUwr$n@3;P9j12C1H1fs0QFfj55d&<7XKNUK_2M`?xTx~ll;00A zQ5_P^d6C+w+W&dO&ZZ`h<=2~mcd4BMxk4D{=WE=2uGeqv@cAx1)jJnTF3rLHb%G{+ zW_7Fj>*V^?cm!?OxVMa-e*ea`zVZckJ(U;LSSup`2&nA09o;|$>kEP>X-+PD*qyk@ z*U;mZ(6u=4wGNZysnqeII&(=4u3p^WZO9tA;azezIuCzqVsWr(G`W!k_`((AhvV#6 zGr4K5+TxIs@HGFMsozUe&!t$1)2oC}ZD;w^Xioz79}B%*khrK_K)q`dR?Uko)V4j> znTo&E(g~YgD2<~&$p~|Pgj>!Dyn9MaL}bfHCJw=8_S8M|(MOA`!;18x?ow&DjvW6q z!8oz`i0h72*WkQsxe;TIjsW@i`nD~v$%f4$??u%kAT@|m^012Ty-U2h=d#qIUek%y z=e7E7_Hf*UHjg@b47By;wYO`;O)hN1#Ka7;E;sCaHG9$0vU*53OVz64iE#3u5eop$ zz%MeQqB5h##uRTK^WN?Ikd~G45Ge4a7Nn*LYz7y>D+Xm}C z$DYr;~{B`oHFvFgqDql6xqnmN|mx~vB zvQ1vFcQStU3aT#=r{nhGHIzLC^0V9tdXsP&08Gj87dB0dRccfPF$p3%YflAyV=jm| zp;Yo(dS<670u`v?Eu3d?EBRJdok=a%dnV(~RJKxq4`-*WJZ|+9JjaIgTd2n${PO@M z-KeuNGL12IW9&nTLV3Eny~0NJUaH%jsmv;8Gj()y$Hu8BqbVu268iPa%FD20*>C`e zL+5FxXNb{CInXdq`uLi;8B{iR7iw`$b(now>FuHmeSxdwjD_QK+y6%IErWqVz+Vyg zNo}dhkThP%tFtT3w*oUv0?3_-&X3WKfV#7I7Zep$RejO(7|2jOV%#yAb1`0eYlDhJ z?~YD>Pfst;sk`kn z$F#x0WTKKm$_o?zP|7ID4z1qSMpn(LBX%n*_WXHWs1p8$#o_oWD468T+^t4)FXQC8 zR^BhQmw3bJPl$;-Gdg z)R9GugN>8dkW;mkj0}q)w%2VCByZwn3Iaf4UN0K$wm&}f+x|2@l{Qj}nkhv%A~*VJ zOs{!$Q5A7yq#&QLQM~18Dyy*}vlUz2>QtKvI6Y&GHn??C9(<^|4v6|VIUCHu;MN>( zoYyz|WUEmk>Bi{GzH_G|>gf3whnVBg(YiWOx;hR^jfQdR>gqK)T9Hr-@)xwXmQKkg zj#u65&cY0}QdHTCzY#j7g86NhVk@!Ck|LbG8I{rgR)1uZRGI6Lw$}C2+{iD7=Masv ztCfjfVOL9Q7uO6=MpbPlF1A`*tzUFoVkIMzBMNA6l}Nc?o25F{WhT4=UcPOG(0z+D zf2X2Lrv_e-n;H!{0urF6w8J^ta?P=VU6odb@~t1X*yLWN=>TUyMvk$NKDyDY))9C# zRll7r+<~Fu{IT1C;0)Jo0eNzTPb6xent9zGF%gqJdvq_1F~c~{;z}$-Sqke+GFsKP8s)W7T;4IUMxj3FRF|PE94a$?zM=5C3bmql_JR8(_=E*Z5 zRZDIcs8f99Cyyo9#O0nO%O~1is&m#=4~^$FechP`F?7aeLK~nB9}}&_cI(WTw~3y= zWQ8ao!l6L`^vVVKNb(ZgAdxn;-)`Ysjkp*(B(jfAgtBETXlv!kirMa@>=oAy-$g`% zHz#m5Bg?UP#FYYpM556goU!65bytV`wfM8UuQ3g8Ja;@NUX)1p+8(T|d!2MCZq1dQ zRilZ6BZ=w#BNm@eFA?>jqwPjW2hAZSH(ewWg9$e(*0>~0(Ok52yiLJ9%SAZB?}EdK sS@&}?{8NPg-cyYK!Xpp|O`H`V!SH!ew9_dc|6mA!D{Cl~D45^;A4#KXZU6uP literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/ancestry/index.html b/content/ko/case-studies/ancestry/index.html new file mode 100644 index 0000000000..a992a284ac --- /dev/null +++ b/content/ko/case-studies/ancestry/index.html @@ -0,0 +1,117 @@ +--- +title: Ancestry Case Study + +case_study_styles: true +cid: caseStudies +css: /css/style_ancestry.css +--- + +

+

CASE STUDY:
Digging Into the Past With New Technology

+ +
+ +
+ Company  Ancestry     Location  Lehi, Utah     Industry  Internet Company, Online Services +
+ +
+ +
+
+
+ +

Challenge

+Ancestry, the global leader in family history and consumer genomics, uses sophisticated engineering and technology to help everyone, everywhere discover the story of what led to them. The company has spent more than 30 years innovating and building products and technologies that at their core, result in real and emotional human responses. Ancestry currently serves more than 2.6 million paying subscribers, holds 20 billion historical records, 90 million family trees and more than four million people are in its AncestryDNA network, making it the largest consumer genomics DNA network in the world. The company's popular website, ancestry.com, has been working with big data long before the term was popularized. The site was built on hundreds of services, technologies and a traditional deployment methodology. "It's worked well for us in the past," says Paul MacKay, software engineer and architect at Ancestry, "but had become quite cumbersome in its processing and is time-consuming. As a primarily online service, we are constantly looking for ways to accelerate to be more agile in delivering our solutions and our products." + +
+ +
+ +
+

Solution

+ + The company is transitioning to cloud native infrastructure, using Docker containerization, Kubernetes orchestration and Prometheus for cluster monitoring.
+
+

Impact

+ "Every single product, every decision we make at Ancestry, focuses on delighting our customers with intimate, sometimes life-changing discoveries about themselves and their families," says MacKay. "As the company continues to grow, the increased productivity gains from using Kubernetes has helped Ancestry make customer discoveries faster. With the move to Dockerization for example, instead of taking between 20 to 50 minutes to deploy a new piece of code, we can now deploy in under a minute for much of our code. We’ve truly experienced significant time savings in addition to the various features and benefits from cloud native and Kubernetes-type technologies." +
+
+
+ +
+
+ "At a certain point, you have to step back if you're going to push a new technology and get key thought leaders with engineers within the organization to become your champions for new technology adoption. At training sessions, the development teams were always the ones that were saying, 'Kubernetes saved our time tremendously; it's an enabler. It really is incredible.'"

- PAUL MACKAY, SOFTWARE ENGINEER AND ARCHITECT AT ANCESTRY +
+
+ +
+
+

It started with a Shaky Leaf.

+ + Since its introduction a decade ago, the Shaky Leaf icon has become one of Ancestry's signature features, which signals to users that there's a helpful hint you can use to find out more about your family tree.

+ So when the company decided to begin moving its infrastructure to cloud native technology, the first service that was launched on Kubernetes, the open source platform for managing application containers across clusters of hosts, was this hint system. Think of it as Amazon's recommended products, but instead of recommending products the company recommends records, stories, or familial connections. "It was a very important part of the site," says Ancestry software engineer and architect Paul MacKay, "but also small enough for a pilot project that we knew we could handle in a very appropriate, secure way."

+ And when it went live smoothly in early 2016, "our deployment time for this service literally was cut down from 50 minutes to 2 or 5 minutes," MacKay adds. "The development team was just thrilled because we're focused on supplying a great experience for our customers. And that means features, it means stability, it means all those things that we need for a first-in-class type operation."

+ The stability of that Shaky Leaf was a signal for MacKay and his team that their decision to embrace cloud native technologies was the right one for the company. With a private data center, Ancestry built its website (which launched in 1996) on hundreds of services and technologies and a traditional deployment methodology. "It worked well for us in the past, but the sum of the legacy systems became quite cumbersome in its processing and was time-consuming," says MacKay. "We were looking for other ways to accelerate, to be more agile in delivering our solutions and our products." +
+
+ +
+
+"And when it [Kubernetes] went live smoothly in early 2016, 'our deployment time for this service literally was cut down from 50 minutes to 2 or 5 minutes,' MacKay adds. 'The development team was just thrilled because we're focused on supplying a great experience for our customers. And that means features, it means stability, it means all those things that we need for a first-in-class type operation.'" +
+
+ +
+
+ That need led them in 2015 to explore containerization. Ancestry engineers had already been using technology like Java and Python on Linux, so part of the decision was about making the infrastructure more Linux-friendly. They quickly decided that they wanted to go with Docker for containerization, "but it always comes down to the orchestration part of it to make it really work," says MacKay.

+ His team looked at orchestration platforms offered by Docker Compose, Mesos and OpenStack, and even started to prototype some homegrown solutions. And then they started hearing rumblings of the imminent release of Kubernetes v1.0. "At the forefront, we were looking at the secret store, so we didn't have to manage that all ourselves, the config maps, the methodology of seamless deployment strategy," he says. "We found that how Kubernetes had done their resources, their types, their labels and just their interface was so much further advanced than the other things we had seen. It was a feature fit."

+
+ Plus, MacKay says, "I just believed in the confidence that comes with the history that Google has with containerization. So we started out right on the leading edge of it. And we haven't looked back since."

+ Which is not to say that adopting a new technology hasn't come with some challenges. "Change is hard," says MacKay. "Not because the technology is hard or that the technology is not good. It's just that people like to do things like they had done [before]. You have the early adopters and you have those who are coming in later. It was a learning experience on both sides."

+ Figuring out the best deployment operations for Ancestry was a big part of the work it took to adopt cloud native infrastructure. "We want to make sure the process is easy and also controlled in the manner that allows us the highest degree of security that we demand and our customers demand," says MacKay. "With Kubernetes and other products, there are some good solutions, but a little bit of glue is needed to bring it into corporate processes and governances. It's like having a set of gloves that are generic, but when you really do want to grab something you have to make it so it's customized to you. That's what we had to do."

+ Their best practices include allowing their developers to deploy into development stage and production, but then controlling the aspects that need governance and auditing, such as secrets. They found that having one namespace per service is useful for achieving that containment of secrets and config maps. And for their needs, having one container per pod makes it easier to manage and to have a smaller unit of deployment. +

+
+
+ +
+
+ +"The success of Ancestry's first deployment of the hint system on Kubernetes helped create momentum for greater adoption of the technology." + +
+
+ +
+
+ With that process established, the time spent on deployment was cut down to under a minute for some services. "As programmers, we have what's called REPL: read, evaluate, print, and loop, but with Kubernetes, we have CDEL: compile, deploy, execute, and loop," says MacKay. "It's a very quick loop back and a great benefit to understand that when our services are deployed in production, they're the same as what we tested in the pre-production environments. The approach of cloud native for Ancestry provides us a better ability to scale and to accommodate the business needs as work loads occur."

+ The success of Ancestry's first deployment of the hint system on Kubernetes helped create momentum for greater adoption of the technology. "Engineers like to code, they like to do features, they don't like to sit around waiting for things to be deployed and worrying about scaling up and out and down," says MacKay. "After a while the engineers became our champions. At training sessions, the development teams were always the ones saying, 'Kubernetes saved our time tremendously; it's an enabler; it really is incredible.' Over time, we were able to convince our management that this was a transition that the industry is making and that we needed to be a part of it."

+ A year later, Ancestry has transitioned a good number of applications to Kubernetes. "We have many different services that make up the rich environment that [the website] has from both the DNA side and the family history side," says MacKay. "We have front-end stacks, back-end stacks and back-end processing type stacks that are in the cluster."

+ The company continues to weigh which services it will move forward to Kubernetes, which ones will be kept as is, and which will be replaced in the future and thus don't have to be moved over. MacKay estimates that the company is "approaching halfway on those features that are going forward. We don't have to do a lot of convincing anymore. It's more of an issue of timing with getting product management and engineering staff the knowledge and information that they need." +
+
+ +
+
+ "... 'I believe in Kubernetes. I believe in containerization. I think + if we can get there and establish ourselves in that world, we will be further along and far better off being agile and all the things we talk about, + and it'll go forward.'" +
+
+ +
+
+ + +Looking ahead, MacKay sees Ancestry maximizing the benefits of Kubernetes in 2017. "We're very close to having everything that should be or could be in a Linux-friendly world in Kubernetes by the end of the year," he says, adding that he's looking forward to features such as federation and horizontal pod autoscaling that are currently in the works. "Kubernetes has been very wonderful for us and we continue to ride the wave."

+That wave, he points out, has everything to do with the vibrant Kubernetes community, which has grown by leaps and bounds since Ancestry joined it as an early adopter. "This is just a very rough way of judging it, but on Slack in June 2015, there were maybe 500 on there," MacKay says. "The last time I looked there were maybe 8,500 just on the Slack channel. There are so many major companies and different kinds of companies involved now. It's the variety of contributors, the number of contributors, the incredibly competent and friendly community."

+As much as he and his team at Ancestry have benefited from what he calls "the goodness and the technical abilities of many" in the community, they've also contributed information about best practices, logged bug issues and participated in the open source conversation. And they've been active in attending meetups to help educate and give back to the local tech community in Utah. Says MacKay: "We're trying to give back as far as our experience goes, rather than just code." +

When he meets with companies considering adopting cloud native infrastructure, the best advice he has to give from Ancestry's Kubernetes journey is this: "Start small, but with hard problems," he says. And "you need a patron who understands the vision of containerization, to help you tackle the political as well as other technical roadblocks that can occur when change is needed."

+With the changes that MacKay's team has led over the past year and a half, cloud native will be part of Ancestry's technological genealogy for years to come. MacKay has been such a champion of the technology that he says people have jokingly accused him of having a Kubernetes tattoo.

+"I really don't," he says with a laugh. "But I'm passionate. I'm not exclusive to any technology; I use whatever I need that's out there that makes us great. If it's something else, I'll use it. But right now I believe in Kubernetes. I believe in containerization. I think if we can get there and establish ourselves in that world, we will be further along and far better off being agile and all the things we talk about, and it'll go forward."

+He pauses. "So, yeah, I guess you can say I'm an evangelist for Kubernetes," he says. "But I'm not getting a tattoo!" + + +
+
diff --git a/content/ko/case-studies/blablacar/blablacar_featured.png b/content/ko/case-studies/blablacar/blablacar_featured.png new file mode 100644 index 0000000000000000000000000000000000000000..cfe37257b99e73fefe25b5f6fd582feafd6dff07 GIT binary patch literal 7867 zcmcI}WmH_-)+NCM0TLiUuoMqnedOJD-`8Vw-yYwO?lZ_OD?FB6*sT2urN0dA2%N#=RJZG?&g3n zb>(z`GyJU}1BRPBL0||7)B*5E(bNp;j1Z^4cly^B>|y_rb%6i-nC=gZ+tn1t&CA8} zXG?zrRaE}(P<#9Tpy3D&@PGULKMKRO++bjC4KN()>|}m_ah42!O2I^=oxr9DsFM~H zYWMdks#!x3P`EV|29TEib83PB78O%-h{K;G>%UkkDk6#waD=IYIapCfoc>;i3j(na zc?sl~2Yw_P)zJKKY zX$X7RePtQ26XZSELe2?l5BS%{MIis^MM~};_5RJZ_-8LX|H$ROH^cqsdjFT}{yTNw zLw}b4Y2N$bKkW~8xbJzV`{r(xaB4(DqheK*k<@aX+D?9mFxGY3815oNW+Zd& z#m*&xcvMeL*O^D>X@|;L3S~juz?0U4#BFkpH>4MNr*FfGSvwC|PjKDDfiVNQTUwq_ zhYas)ihWGI?7kw7AiPMVAXklp!?;J|T2@>&ibL4TfFJ6tJvLb+yxDv9i*7ADo*}=$<^d1aYGRXb3+!c zL2{gyWbl3kTY)-#5E0blU0YXVkD$7bV!9Ak+1&of$5@aOIm(Kq9)9-tBUW|Y`sFJ= z?tL3&EVYHPFItL2;Dtv`uM;tOyZL9Q@9aC(2DnrS7LjA}Id14=oZ}$DLXGoX)t&mL zj_e3WnKcDQH2fUpcc8#R?1-{X^BI|lvjvx#)Ry_>8YZcaAxu8ZashghjQAjd6Jz?W zS9<3Zm;&@HzLZzD$j*}56Lxx5xm>XmEZ{(SS*Il)&rbgvL~4|X`0J~qeDVCnroN!| zA%+dfUcQBcMC;io*O8^EF`&v3)8Px3_LHM6FS_FgE&KXX0r|cM?kJoDnv6KWArDfm zf{T2(L(pXVX56qxKWD-1m8m1Pg(JZJ4aIJEe)O_a`20)A_Oot+?9d`5qR=(v-T*;R zDcvxoZi!)xkx6NqB*DQM(MZ24%)=hG{%JXr+8Kas+3DuP4w(ERgx-b=VMw;WqpfWB zQ`E?4vWGROc?1~)pRQONdm1+;QuGm8wYaiIE-d-bva!_JT3jARrwmz~hcG!8H^c1d zaa&U4XQb-V{mt`eHWtJDW@nchH@Zo_v-c*2Gj)k9>IZyD5gOYQ=EfbqtaF>MIYiYO zZ1ZE#mK$-Fe4+1>U7&?*epc%9z~|vIphM!0&BcQ`GmIsaGxeIbvPuZ) z-)HKoFJbZdvLSD-X{kerik$HmZc&3uBeS~;sN%-zqZ9;4r(0mC**0S5ENkP7%dAWy zx)~pvc6c+}OfB9-^mPU689EcAnu*o~X=RtZ>oX0EUczW|dC=W9NF276;v6M1{c(NV zdZGM!Kf6q*WPF&k{?5<1s(`eW7g9Gr1G8w`lCCfbRbH5UP8AVPae{#abgyE9xX72yP%(L1eMHry#$}8_=S!#Z@Wx zX&XFmU2WQ544AT=I`@8;{f}zkWPqLzl`JOw_j5 zFJH4AI#=_%eYUOaGz&40K~bS*l1m8)*1?2Ef@1i%M?&Ie$TTT!s~)uZ#1o<(ncz3| z!#z$NVid>p)cl__R9%LrFsJJOa)MjUZrF_xr3Lm{jO$%%5z$8<7DJWR`_oKU$p(Vw zysb=cuBJ;~#D<)uZ$N?GBOD?za^? zwB2jhon)>Qb`cVPQoHwAjb3e};ufBCwcR@-1M+U`zs!Z+&{Seo2eK6Ua}$w^%KVW#q++HFoKtHhX>QTV5zu0rLEAA7VtCSwK`_=l++hfO%wA871>aqmaD>_dv| zf<7+J3rAf|mZa~_ky1&c73blqZqrH-7>-tKC=J3GO$ZIV3~F|`tuPBc9#$J$0Xy0U z+(sgVs9c<`-ZF-=Ip%<46ug2QBrNT{5^qn&R|Hp3O^;HA+rGLy`BSVp1_eP`5ql{ z*8FaTop6`c5w)Pvx+DB$9id&nU zd+g^WRMSgopO&)-HfNq5=~llTQP+gzl8qJE)^Fae3^n3}D+k81z8Vu$FB2>18+LK5 zyAp2xUeK(ev+Q3}_r5==jw!BI=7o!)u)@z*gZP$ zo~{GQ-ckg^v$Wu9!6P>KAdg}1wMeuP>n~r$xoCyi?VLBOrp0sl#HUNN!}C(l_T*72 zp{GIa=sCnS>c>Akb-lK;1S4ByV_lb+8U;lk)mqejX=q$-1tnCQg-iG;o(UJQM}7?; z%i>|d3T4gbGuLLP!^e-hJdI2yE_=Gdx(h4d;nq3Ufb?EXle^pXV&M~Epq$25$x(A& zlG=&Z!;{R)of7jtTE-`=TMovDJ zOdu%ogd6~z4|9{;Rfx}D3%;(EcOh_zSOZF-h*nX~5h6@w3uZcL42=w{&YaA8rhrk?!)Y5zNq);0$^%c>8=B399Noso9;S4aDM;tc z|0ExHd;VSoxx;x27G$$6B^O;!w(ID;6S})7jc&Clua`748YBck_n#DRul5?Fr8Y6HV@$1B{_w4*g+OB(Cy%5DR2y(TC8y0!lPhY&o zdceLV8cyx?)XY#JcRjv<2jjV+CUdI9rNP<3Y}1y1KL*R%qGm|(_&Z`5;N|dn`LAxb zN`eV1ZOe;(Iz;1pYcs<$J+;YTJ83#hc9yMSl*?f6jV@}u2%;J@8~Q`=*M@k4mUY)V zz~uF#6vpXX)#s_MHhW#WnbD!rJ(PV#rPwf>Dl5k>eq*O*pX$1q$S_fLZly@w-@yBS4l+o71gel_cw-OZHY1 zJh=w??0}~3w&r|1B{M%C?MNvL&7SLyZiR2WseD>eZd~S> zR`BFm6WEeO+a;JyJ~(K zy&TU3->w#kfBkC|GHxWaeqDRu|KhfknNY}77kz=3h6Ue}yZxaqrq>3NU60wRyH8(+(+cm?936deOJ6Oq(3v3}MR^<$j&-!MaW4MK=X9nR~(>H9KtvMkV@Y z?T3v!;)b%Ecv|Iz0SWbaw*6K}BNJ_m0MAh~(n!sO;3ACd?|n^KI&nm zj&062rS|}uy&t$t9|lH9d=pIqCS>Dt;kQn3pN9F{84C;E{*cS1xcV+0(m@{A@IGLbrC{E7}OIh45T0u7*{9E1=izl~nQOzY);kzuP%E zj_sVp#(p@1KZ?UY=tatpcY)c8G<1{rs1}i$O_2FgXoQ2<#E&gvJ-3>I&I*i|pqX~j zepLGDYnytrMTT3Mu;>UwddgBuTEGX=vXIxuf()LrC^3|e(5(C^%~nVV(lf6&Nu=>` z=;5Y;enOwf%c8Iicdzb3DLD*g!e=`xluY#Vj?SMn9e0U}hQIrpVsRU3Wum8yK=4D5 zEglCngeqj1>hW`>{(GV}1uVLHojF|@@~30 zVnkJ+=JBqnX7uvYMe{WE-COsS#l0W2`net(FR4>Rn8=$)?67EGCGGS7SKW)r#VPw=*c$y}ounIoRXs}(<)6P4aQSrk$@7b}J;!Ay5 zD0!^s+^u@X_K+7o7D_Uinpa~*!Q?RjlgBS&q~Or3UEcTl;D^(|Ery=(EiSj7` zQ}s=HVmxzIgfO>}T(4Gz~Ys1f#fcG32)l*jkM)UE~MkkC%sbiU*ePL+|76y;Uh-6n4yz0dg zV`f{L1%dWwhGI5`d|0<`i!q*Gx11`E5&v*v<%z1wu`EB5vK3e1AiPw}&Zc{^TzKYm zyFnP{mYILe+-xb{T>M)j@RRQk%J3clxpBw1_S2-f;PUUiy3?n_=WGs7TTi!*_Nd+5 zeiq0ak&QpD=|WUcB_)bwe_x;#t@BHHo-C1tR`O+;TX}#}yHSkwM2ilO@aYTcP3gQ5 zOh-xE@fPV)g~t;UWTgxB10>V5A1X$*67bZhMv`lQNhPd<}J!$V%q#g1kIZMe(>J^5;4 zjZ*J6m#@&UgG|x#%!MjmIAvcFY1p?Qu0}cz_1%pQbH_!_Zuj2#F>$!?n>||Ftubsq zQ}XZ#jOw4O8@xRQ`C2inIJd{_d`dOu;&NbfcCsEOMd@~J?A6r(*Yq+Bnut8=fBv}C ze%EH)=IkD}J&02Wsygr?KWjw0^_C?e7AyQJ6YXMj;z&^*opfv;^m&_VM0B>iCW}k6xxsj6DwXC-q{~Fw zE_FN)?c7VJUX0SP2y*|Bmo1N9K~2W=LULq$50F#F9~Y1sZ(^{V28)#@53D2J2$n0v zPqG4Yi;_z1t`7T+_roHA#=QZbj-s&~PIAHI6RW~;%(-#qcMV%60W>OJK<)sksR>qNG4%IEvot`0dm zKbpW%`>I4DUF8j_KM(8)ACmPd)^>_S(B#+c)($gZ9;<_E+q`!FuG(bw_Dm3w!@1{^ z(^X|0MoWrG?|m_IzBH058Rl^Y#jEy4_cedqcnxshNCqx~F27#H%dJu{7yO7{xMm&8 zq#w#CWqYRW)Gm9RZ!8(`8^`a-cKu#?(jWs(@o|@h7^vNn2VS z@dk=6YgD9!J3{6Qka7aO$!_NX{wAfq-k^8G343rzSi7uyE4=S}9sa&flkLu+ijl!I zcEoBX^Mi;xBN^GEaU^rrMSJIX9hrs4VrupNO-^cl% zoc9bH93dlhaUUYW)=Y^rc)_a`9aM3Q(ju9Bch_mf$hpH>HHjzc&G6<6UGM$^;rB1- z74rI`UQ;^<0Xy^;b-wE032%pz$8j%}cyC``kAN&ya83`z9jS?vDo<6XR7Y zmwk<1r zLb(eGToboWgb9|U9%UC!B{FCi`}?!WkxUh|pS1LLY2&L(_&KY{FC~V@k_56>M3;O+ zEn&thbR}>l2sP__FMJ4SzOiz6_qJ><8%eM=fSavAlKQoJ0$N`Zsl&xKoL;!uc$WC~ z*6fR3)w>U$h;$~wosXK`xS2!7bZ;{#_9r&5lct#368kato)L~eAa}atTk5aJLpo#k z&r33#1rXdVO=67&J1#aa>*0qP>kM|ii<&^t@z&1xE(T>|> zoKA!TuAq^Z2ZA8c?>JUYt<9}y(gb%!$ykDZl8dPYBN zbA-U=)@FL#-6D==UX)w`2L$!LnlW2ZhcoICuZShavm3e9B4(L}TgRg7jr^9^nk70K z%b!1DHCGCb?B&u0If=()%qjF3-H}H69ww~&Xzwfw28jM#Se#KQ{nVL~zx7yq{B9}Y z!5I7pQt1YCvs%}>@-d6~yViz*QM5%68vfQadES9zVOdsFvC2@i4Zr%0JlS3hieO#h z=qFd%bjSMu2C%egG|o0~PhW^||aj$Rkkn*{6=}9h~5tsG!}5 zM@RPXc3JojVFK#Gw9^3L^G;<;Pv1Mv$HMY<AnB} literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/blablacar/blablacar_logo.png b/content/ko/case-studies/blablacar/blablacar_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..14606e036002ed9ad34469a06a8ed39f86e41d71 GIT binary patch literal 7841 zcmd6MWl&t(wk{f6LI@JvokklP_XO7<8>eZUrlD~SF2NzVYaqC#3GNOdNaMjtf&~j6 zE;;w?bKk3a`~AMFYOOivSmXPK{+LxGT1(?40qzT26ciK!1G_l$ zTK>`D^>&6mqES#Jq`hI5U`L2My%ogP9xBOr+}6cNZ*MKhXdtZ4rw)^c*xA4Gb%W^o zYUqJ|9l;=LMrkQ}32(7S0%wT3CB3(^6BI7yEy?&VUa`mhA2EQD{$DEYj*^W3?UbRq z7QMWS8-!k%7svzV7Zjxz0rBz+gG2>{xakG>_<;aE5P)BZhmT*34Ag_y?Er1^c0s;6000II$j~YC1AE>*fHxCrf z^f!Y71P*qyhq>FkKg@ad|(iO4g~Jv;Rb#zoDI_-Q!p`kH;ARXi<_Q{i__m( z)UtDNcY)iv!06@W|71;so?YD%Y!Ce-bZ9j`*UV<*(LK(<7DT>wE8%a z@yYtWtS}a4Fm`YlXgd)_GwE>TUEXx4U9B5HY4t0qq)d$|X;& z3>DoQpY)wMXX zpaws_jp)0lA!_P0;z5L6Rf;AOdte0yi_7)A^p`WPx~f*Ol>Ku4uvmTyI(>XZTD&v* z38N>fc1@cL@~nEw#C3E}`m49qiL6Ft)>`uefs_`Zbf2K9@t#!OD}*odv!FQE?dD%V zuw0mpk+pGrrm#;mIrG{qH*$b_NGu*Nqw^Ddoi_M>V(u-uKGt9YF$rQxrMXdj^6mCK zC=E6iUd3;Y&I)x{` ztt%5~0_z{mn9fj5595E&wjAESp${V(@(lJ#1Mei_zv^Kov6j_Qq$labi$QVKKpuVw zYAls@COaax+SPwDID+m#uq+9bf7fy8dNyyIp+FR>@B~u=VJp=h?f9M` z(0HPj$`J?3YUT@PDMnXu3NPDD@lABo5!uilnrNxM&+8`*76Uo3iG$Od9pM?wb70Dk z_V3R%;ilbU?c@e}cRkl)W#}H53JUSI8z%nIl^N=G(*C7q>Ez^YCiAXukSjFA*K1)? zm5Y&V&WbX4^^NLP%gY%R*@B0*d<0DMM zhdaJdo~Q)ABxSN*mhM6s0myDC9R zflMmenlsuYA6rFB+QDdlR;c#Fb2#UZ{1)|o*EJ_4PoVZYEF?0mzk8{(r6p-n;ea?% zy`cpHD~d7P^wBAnT#dgVVUsZ7Ka8|B0lmhd*$KlUFvd}3kMxJS$w=UZmsV&CGoM!H z)*obWU@9CWVKbQNg-zTlaeK8SmIL4863gD_g9sr*v8mbx#Ouaiv+tHWr+HSR4Rgky z^@@nnM%#i;Cs(nov~cf^<|bJtrf;RaLb^Kx?h7R^yzXxL@QwFZ&DEf!K#LDqN>Zlfl!-tkZV`vs`Z~Y82I?zkf0% zcbRVSj_nPr^ig8PPTWbbE3Ppb#B6qZaFMp*eABBn=+-=f@CeP{NF!_=?|b$RivR_4 z7NdSxzk-{t(^a=f{=+qT`iddLa7ymaa{){TS!56ge(~Y#T-Z@-Q>h7=f2`G~`(j}l z$F&3UIg=)Iax|HE>*`Mh`Bxd5LaCC?zur)%-YeTuX(({63|dZnx*m~K8r)Wl@5Y@g zm$RY&^@HGwX!FC=<<0zWN4~Sqkh6F~yh(@2P%r-X0(5Ao5QKC*Y>Vg1cRZCm-xEsD zm-(Fz!ZY_}2Ew#?ly!%Q`c(0(I!{k|MY$uiL8pM{(v4{!YGgJQ;y+K}tqD?g`2?#> zB5B4L73CznR@n7KHAr5^Mep1*?NfYj-MN@^<4d0=_8N#~2|8SRK?lmvoWKq-(&IiS zO->uRS@9+x84Hfqk$(LjAo^l~ z`z?Dxfv0<{;u}8N&qrm6;;Ah;CZN249JTQAhi@T9wN7*>Dq2!$C~}Mj6w?A0gybw3 zd8_0|&DU<}E5yl_WADr+LL^XknA*vHs1CV}O!FnZPn5yydNMl4vrH)Y5ph(s8&mu) zv@4K|H^l^eWJ(j|`6c0)SR+HK2R4~Q(!Mu%(C4`F^2iNKDf3#;t;I$&z4bi8I6G8#&Xr219GlS6~{m*$1*raVjmR;@WD=jX@aDBO(D zrQ%FOJcY}rJgP3-AK;Ei#mcO4>L?d6%A`XR>8WPF`EU2Q8JxY z6wFtfw((}jG_=|3Xm~?kbF4QrcVOV|Dp5U)IfGO%GKw%<S}hX~WIZtbYwP&d9Z9 z0W3uyu65e%7c5lP!{VA;my&NjXmTs-2y^;Go_PQGON%|VeXLS-S15udI=j10p4IL> z?t%T(9AA5P=@%Q9Q4|@rqj2X{^Ck7Mq7ontqZFGHp8l^Ar$LRhD)BUkolFII?NiVo%(_g;zWisE~|J2!H(#e1&l z)$p)^?y^)fm!y0kK3i4}&LLr)Yu(2Pc_%G-%VlXMA;E&3OgD%fPMuCkHL+mBe4P5B zG&z1SDswfPdQD+Ax=9Fh0SyE_PB~|*X!Mg>KanT_>#ZtOMLb&ud_okz`3SWHpKev zO*rf%146g*v6DfswZSr2uYa^?v@HJKMW+26E8 z3ZEBw{tHSDz%r-#ZbX&~T*PVXwiuThNm9PaU^g=CjeZ)($F+i>TSxqCKuTRij_cJy-(CUfhSZ4> zR8|lV_4U)k_&`#vc)@U6eYNY*Qh14peBkATZOGUV&_!T}x}v5Oco^VOykBMi6?U(j zFWJg%AT=MO(|Pl2nbOGsC|o|=WA~T?D!SvNRdwztdU@O@>XLU3bw_7(C2?rD}Mv`}IlD51t_8g4}DI5-DF1oL)i4caeBxc9zlQkJ7ZPuAA zl$6U(kgJZ%y*m4i*h#~*dV3p)@cyTCnMplNXFc8b9~+*BgGS~IX!ZgN zMYd=Zvako_-_R3(DjN(1%fM!M9%_eG`^|Kp3`Dj;gDmIdxq^sKsYjwvHRHFJwTagj zJ|@d=k^ZPT12_i0qdwf-A*Lr}U78+xbL%rvKFR(;yrSX6tIhoAV0A~BkPH*W zW+EH)J=W~#RI<5_mr+iL%uObZOmZ-OKP@}@DOd5pE!GPDkmo0^blm4EAS{zH63qj4lcUn?jZU_R<;;Clc%vl#GuH@#rz%_$Zu^r>HiCqKFC-inMuc~FPafe194eM zv{ddX68kcMHkAu^ke7e;;!9PTN;8oCfMz~#>5DkfM&{S~@tS^g+t(LH-(6V3 zn(~fU->d@N{rH@3NPB0xu6}Baxz7qZZCN!MTChQT%}j>81U9JV|QCyPnhPQa5Pb&{GKxGzhvb#gRn{6Z6j z_7zNf;k^I%htjK&60 zokKCQeZXc?T4D8Gg@B1I{8q^f2Q8>1p}0J~g}VznY0vU|>eynBF@-`izKoksfReLvAO;rHoZO(q>%cge{QGlCghdj_T_pMUA8u$7j384peL8GsUsm zL}p-N_Q17QZ1d)`j0%COnQiLy%x-|Wsr%AwNDtDPe3N}_aZ7b>|3)W$sy46Wf}EX_ zsj5vr+KXQ;Q=E3NR$7)UGtodic&mr^oJ47r5;=s)8qFx z-hiCftms2q%Fmn(%~`#|0_wSk;9L8%KjFGYBmRj#ErqOhRFeqFA8+kba(R#@$8p-f z-LyUj{x0>(sr_}a?ve1!cI()!TdYrI;GS!s04IcJ*cTVA9*E30#GV2zO#cNCF%y`yxIJi`!`!vOT*+$v{Olu;A2@Fr) zE(H5mymjq2K8yNkWr|o5zVPp!DY=t-hE*T(d~oPp3cAaI@q-rfS4Qa&&sKD(UroRU zIgRPEA;VX0l-m*9>J#N5<;}$gfzlF!h+(A+rhJeMsKqEMoLCAH9mi?iIrl_S7Q@>O zYkFwB!9U@X$u6vLqUR?`&~pA|JB+?Y9N{&NumvBNb}i+oQBs@piG*^1ok&7^=L|N7 zooc1i=>Vjkp`9tN2`v@VlrgrQ{%=n<%S_^gJ|CBQJX8rao$Iv}_1lR@Mo4<;I4GR% z83xAQRV#}8{>KC5w#`qj`lQlf^^zXx)ZN+b678CwTWue`f zO_0t(fASg^(kxaM1zQQJziZf0-N19i-l7n<+3yJTjQtI+|Rq3Uj#|J8oqowlWyM; z_}b0!YG2hnDBE|>Yw;(B=iCZntbNONT~13~vBm-xK+d?UBta`JvBE)O#AGv@0OO`^ z!8hX_=RuCoC;q0Rz%y>F*5J9=TWfJdO8p$IN$*aIP1n%}Hp6xshDCkayUEft`<=vq zMSE!7BD=W+I7n0~!q>r#vJ8^JD&bjwn34NiU9U4i(^<#o!}iU(vs zp@lNAFFu6(wEmN1@zM{PP;%)y4{PSJ=Z6`Q1^(0RKm5a#&~jWl1F-9$!?xB*@*m9w z(*3==d$!bNw~@~&_Q8XSwn8f~ZM?gJQupP$jSV&KF*Vwyl17dGq*i>59jq~yu~GE4 zM&w#fz!giI7q&6*`wZ*gxv;UI$O?EZkPD0s$0a#q)6BwI5p}()j@vD3={;Clv|PKz z`qGd|A?1(H?oVe`$oZsQUB8v~*$J7X_;TY&;T40fCOAspH!anQ8P}?o3VnkUFaFB- zY?4-bxV>^P;E-y#3?un{kEONzHIa<^YdK`z#T3!q{bl0kLf&()?;x^qQxjA1+sJZr zZLW$S?hlz?5$xkQF9sXa%zSMordx8}ppri?q^K?T!iS1xtSsj8brXOLr(_1hMvG9m zFzV7k@5q5K&*nIFDYdPNjV6gAunE}kJmz8shcRt|o$c4e*2c9c7q=n6yNBe)7eQ`; z@Lm}0dz_qAox?W+8;dh#5w>(5Up>~>j%{l@pPt!Gkh&@Ooe?rX+&A`$ z3bMlgerf^)C?Npnc_ z8B3u`-Hk=~J2*t6V?6M!49@2}TYrGAZ4(YGle7RZ4ae!AQQ4Y<_32O_|JbnRrDES@WcmQf|% z{#X;0niFQ?A8qCwb0y^|5jZ`GO?GA%nD8R0A=#vWlMiEILEZo%Dx6`juf-4hmr*G! zw~p=Z8Y;HIJ!AqAlzrxC`lpt$2eQ@zt&J(W(Zjc)WPmeoV*yZ7s?>1pqO#W$S?yOhZ}4?>MfRxP^5#&oFY1X{ zQM&n(pdq&fLZbsTf7%>QycVo651kA?B9C#u2Qv-DEHeRsU?61&DOrpoRwp~^*x z#nrg6x1Q{$*c~%HJLSkqqLuXfDRaJ#HVJ{#IG!(ko2S!b56m&Lzn2>1428Ick~qh_ z)+)aKqybcOty$6R6LhEp$A5p*h}etE?##n0vM!H}p76KKRq-7<*F=WA`1Wn^8$#O- zk&!MRpQ(tyA9NJ$#JsJDg)4rLNFZ#@O#{;6Z}Z(fwLeQY5UGEO-wP6+V9UL|5MzOz zPgnfx{o%JV34EWVnRdkz_Rz{!hy4>$LXZJ;lX2sY+*$ac;t?3|vvNQ;Z5}g0V)!|m zeyxd?X>sQSXKzw|jPE6jKZM!)>t!6$(-2SvUTP7jQ3|13tcCEi%jxiKtq`#LpbRKrV`pa)m0SvZEQ zL!?U{;uEsR5%+z&yB)8Rv}HW7uofaJ?y5NMQaK|a2{*k${z+B{f!@l~dTF>esWdUJ z>=Z%MOCQuW!YNfa;~eM0%kzzUuZ)5aKZmcwD{jJ;hSLQ?ALW9xLLR>PTOBf06D7)V xR$j_6+^287kMH_lHOl$FCrZAyH$kXRP=F@Iv&I@2+kb|}%8D8aHF6dq{{@bPm4*NS literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/blablacar/index.html b/content/ko/case-studies/blablacar/index.html new file mode 100644 index 0000000000..2d55ffb8d0 --- /dev/null +++ b/content/ko/case-studies/blablacar/index.html @@ -0,0 +1,98 @@ +--- +title: BlaBlaCar Case Study + +case_study_styles: true +cid: caseStudies +css: /css/style_blablacar.css +--- + +
+

CASE STUDY:
Turning to Containerization to Support Millions of Rideshares

+ +
+ +
+ Company  BlaBlaCar     Location  Paris, France     Industry  Ridesharing Company +
+ +
+
+
+
+

Challenge

+ The world’s largest long-distance carpooling community, BlaBlaCar, connects 40 million members across 22 countries. The company has been experiencing exponential growth since 2012 and needed its infrastructure to keep up. "When you’re thinking about doubling the number of servers, you start thinking, ‘What should I do to be more efficient?’" says Simon Lallemand, Infrastructure Engineer at BlaBlaCar. "The answer is not to hire more and more people just to deal with the servers and installation." The team knew they had to scale the platform, but wanted to stay on their own bare metal servers. +
+
+

Solution

+ Opting not to shift to cloud virtualization or use a private cloud on their own servers, the BlaBlaCar team became early adopters of containerization, using the CoreOs runtime rkt, initially deployed using fleet cluster manager. Last year, the company switched to Kubernetes orchestration, and now also uses Prometheus for monitoring. +
+ +
+

Impact

+ "Before using containers, it would take sometimes a day, sometimes two, just to create a new service," says Lallemand. "With all the tooling that we made around the containers, copying a new service now is a matter of minutes. It’s really a huge gain. We are better at capacity planning in our data center because we have fewer constraints due to this abstraction between the services and the hardware we run on. For the developers, it also means they can focus only on the features that they’re developing, and not on the infrastructure." +
+
+
+
+
+ "When you’re switching to this cloud-native model and running everything in containers, you have to make sure that at any moment you can reboot without any downtime and without losing traffic. [With Kubernetes] our infrastructure is much more resilient and we have better availability than before."

- Simon Lallemand, Infrastructure Engineer at BlaBlaCar +
+
+ +
+
+

For the 40 million users of BlaBlaCar, it’s easy to find strangers headed in the same direction to share rides and costs. You can even choose how much "bla bla" chatter you want from a long-distance ride mate.

+ Behind the scenes, though, the infrastructure was falling woefully behind the rider community’s exponential growth. Founded in 2006, the company hit its current stride around 2012. "Our infrastructure was very traditional," says Infrastructure Engineer Simon Lallemand, who began working at the company in 2014. "In the beginning, it was a bit chaotic because we had to [grow] fast. But then comes the time when you have to design things to make it manageable."

+ By 2015, the company had about 50 bare metal servers. The team was using a MySQL database and PHP, but, Lallemand says, "it was a very static way." They also utilized the configuration management system, Chef, but had little automation in its process. "When you’re thinking about doubling the number of servers, you start thinking, ‘What should I do to be more efficient?’" says Lallemand. "The answer is not to hire more and more people just to deal with the servers and installation."

+ Instead, BlaBlaCar began its cloud-native journey but wasn’t sure which route to take. "We could either decide to go into cloud virtualization or even use a private cloud on our own servers," says Lallemand. "But going into the cloud meant we had to make a lot of changes in our application work, and we were just not ready to make the switch from on premise to the cloud." They wanted to keep the great performance they got on bare metal, so they didn’t want to go to virtualization on premise.

+ The solution: containerization. This was early 2015 and containers were still relatively new. "It was a bold move at the time," says Lallemand. "We decided that the next servers that we would buy in the new data center would all be the same model, so we could outsource the maintenance of the servers. And we decided to go with containers and with CoreOS Container Linux as an abstraction for this hardware. It seemed future-proof to go with containers because we could see what companies were already doing with containers." +
+
+ +
+
+ "With all the tooling that we made around the containers, copying a new service is a matter of minutes. It’s a huge gain. For the developers, it means they can focus only on the features that they’re developing and not on the infrastructure or the hour they would test their code, or the hour that it would get deployed." +
+
+ +
+
+ Next, they needed to choose a runtime for the containers, but "there were very few deployments in production at that time," says Lallemand. They experimented with Docker but decided to go with rkt. Lallemand explains that for BlaBlaCar, it was "much simpler to integrate things that are on rkt." At the time, the project was still pre-v1.0, so "we could speak with the developers of rkt and give them feedback. It was an advantage." Plus, he notes, rkt was very stable, even at this early stage.

+ Once those decisions were made that summer, the company came up with a plan for implementation. First, they formed a task force to create a workflow that would be tested by three of the 10 members on Lallemand’s team. But they took care to run regular workshops with all 10 members to make sure everyone was on board. "When you’re focused on your product sometimes you forget if it’s really user friendly, whether other people can manage to create containers too," Lallemand says. "So we did a lot of iterations to find a good workflow."

+ After establishing the workflow, Lallemand says with a smile that "we had this strange idea that we should try the most difficult thing first. Because if it works, it will work for everything." So the first project the team decided to containerize was the database. "Nobody did that at the time, and there were really no existing tools for what we wanted to do, including building container images," he says. So the team created their own tools, such as dgr, which builds container images so that the whole team has a common framework to build on the same images with the same standards. They also revamped the service-discovery tools Nerve and Synapse; their versions, Go-Nerve and Go-Synapse, were written in Go and built to be more efficient and include new features. All of these tools were open-sourced.

+ At the same time, the company was working to migrate its entire platform to containers with a deadline set for Christmas 2015. With all the work being done in parallel, BlaBlaCar was able to get about 80 percent of its production into containers by its deadline with live traffic running on containers during December. (It’s now at 100 percent.) "It’s a really busy time for traffic," says Lallemand. "We knew that by using those new servers with containers, it would help us handle the traffic."

+ In the middle of that peak season for carpooling, everything worked well. "The biggest impact that we had was for the deployment of new services," says Lallemand. "Before using containers, we had to first deploy a new server and create configurations with Chef. It would take sometimes a day, sometimes two, just to create a new service. And with all the tooling that we made around the containers, copying a new service is a matter of minutes. So it’s really a huge gain. For the developers, it means they can focus only on the features that they’re developing and not on the infrastructure or the hour they would test their code, or the hour that it would get deployed." +
+
+ +
+
+ "We realized that there was a really strong community around it [Kubernetes], which meant we would not have to maintain a lot of tools of our own," says Lallemand. "It was better if we could contribute to some bigger project like Kubernetes." +
+
+ +
+
+ In order to meet their self-imposed deadline, one of the decisions they made was to not do any "orchestration magic" for containers in the first production alignment. Instead, they used the basic fleet tool from CoreOS to deploy their containers. (They did build a tool called GGN, which they’ve open-sourced, to make it more manageable for their system engineers to use.)

+ Still, the team knew that they’d want more orchestration. "Our tool was doing a pretty good job, but at some point you want to give more autonomy to the developer team," Lallemand says. "We also realized that we don’t want to be the single point of contact for developers when they want to launch new services." By the summer of 2016, they found their answer in Kubernetes, which had just begun supporting rkt implementation.

+ After discussing their needs with their contacts at CoreOS and Google, they were convinced that Kubernetes would work for BlaBlaCar. "We realized that there was a really strong community around it, which meant we would not have to maintain a lot of tools of our own," says Lallemand. "It was better if we could contribute to some bigger project like Kubernetes." They also started using Prometheus, as they were looking for "service-oriented monitoring that could be updated nightly." Production on Kubernetes began in December 2016. "We like to do crazy stuff around Christmas," he adds with a laugh.

+ BlaBlaCar now has about 3,000 pods, with 1200 of them running on Kubernetes. Lallemand leads a "foundations team" of 25 members who take care of the networks, databases and systems for about 100 developers. There have been some challenges getting to this point. "The rkt implementation is still not 100 percent finished," Lallemand points out. "It’s really good, but there are some features still missing. We have questions about how we do things with stateful services, like databases. We know how we will be migrating some of the services; some of the others are a bit more complicated to deal with. But the Kubernetes community is making a lot of progress on that part."

+ The team is particularly happy that they’re now able to plan capacity better in the company’s data center. "We have fewer constraints since we have this abstraction between the services and the hardware we run on," says Lallemand. "If we lose a server because there’s a hardware problem on it, we just move the containers onto another server. It’s much more efficient. We do that by just changing a line in the configuration file. And with Kubernetes, it should be automatic, so we would have nothing to do." +
+
+ +
+
+ "If we lose a server because there’s a hardware problem on it, we just move the containers onto another server. It’s much more efficient. We do that by just changing a line in the configuration file. With Kubernetes, it should be automatic, so we would have nothing to do." +
+
+ +
+
+ And these advances ultimately trickle down to BlaBlaCar’s users. "We have improved availability overall on our website," says Lallemand. "When you’re switching to this cloud-native model with running everything in containers, you have to make sure that you can at any moment reboot a server or a data container without any downtime, without losing traffic. So now our infrastructure is much more resilient and we have better availability than before."

+ Within BlaBlaCar’s technology department, the cloud-native journey has created some profound changes. Lallemand thinks that the regular meetings during the conception stage and the training sessions during implementation helped. "After that everybody took part in the migration process," he says. "Then we split the organization into different ‘tribes’—teams that gather developers, product managers, data analysts, all the different jobs, to work on a specific part of the product. Before, they were organized by function. The idea is to give all these tribes access to the infrastructure directly in a self-service way without having to ask. These people are really autonomous. They have responsibility of that part of the product, and they can make decisions faster."

+ This DevOps transformation turned out to be a positive one for the company’s staffers. "The team was very excited about the DevOps transformation because it was new, and we were working to make things more reliable, more future-proof," says Lallemand. "We like doing things that very few people are doing, other than the internet giants."

+ With these changes already making an impact, BlaBlaCar is looking to split up more and more of its application into services. "I don’t say microservices because they’re not so micro," Lallemand says. "If we can split the responsibilities between the development teams, it would be easier to manage and more reliable, because we can easily add and remove services if one fails. You can handle it easily, instead of adding a big monolith that we still have."

+ When Lallemand speaks to other European companies curious about what BlaBlaCar has done with its infrastructure, he tells them to come along for the ride. "I tell them that it’s such a pleasure to deal with the infrastructure that we have today compared to what we had before," he says. "They just need to keep in mind their real motive, whether it’s flexibility in development or reliability or so on, and then go step by step towards reaching those objectives. That’s what we’ve done. It’s important not to do technology for the sake of technology. Do it for a purpose. Our focus was on helping the developers." +
+
diff --git a/content/ko/case-studies/blackrock/blackrock_featured.png b/content/ko/case-studies/blackrock/blackrock_featured.png new file mode 100644 index 0000000000000000000000000000000000000000..3898b88c9fa43c09caea23dd985c1bab1802d986 GIT binary patch literal 6391 zcmb_>XIN8P(>4k!N>%9~B_JZw0@4G)5PFj$y(JKe5Q0e%LKW#vM5+Q&swhpQBSl1d z2LV9==}2!81PSl83mg|HC@AQ(G~q_T zvlIBHQJ)2V^_hCjfCoEX%^Yuxal(5euqcXKju<40R|}1BMj4?Hj)Z%iC*gTu2!bl}LVOee1!xo=!Rv!|_rNLmD1rXas{pircZ-8~|6swp zDS`eoDRaHsys8*1idR}pM$`d(Q-W7kUJNWFaZ?&B!Ye5OmKB$f7MG9_l>jS9$SFvI zdH?)?fYGpyP6|eF^*_e~-jqNtc)X{AxVX2sx0tt-7zXPs4wjdf7nhI}my{F*I7D#- z4?Mz0)C0%=mj*Zr=YVzf#JgfVczboe=+Lm{r_3f=)bvfcq7z*=lhR}aV7*$l(-QJhw;KX00-y9|9dD;1yw8xfyZD? zFc|m0cJa0g29LqHU_5zMRe6Q<5Du;$zu&L_A)u$Hpyh$XBRm{XT5u&0phC>m)luQ5 zw5l9TULCHcE-eoRtI5j2q+qHy)xc5`lB$x@a%z9c!Z8kBXp{&3FImU`%BuaV>~C8@ zdjccFQCQdeC`WZH2F?4&niX9CeJ-$n)%T~Yy1wdHxVMRdVKS`a{n2hc84PGEq~f1`wJU zaGRM+Tr<82Zlsf@>HZ$}4W({eU%7u!H;x#CA`l40XZUh&6@8`(y)E$1#26j=&jd?g z{AUun1p7ClR)hUBWoc6XBk?hv|5pkH@%@M9d0r5V#fZ{wrq;T;y6H~6&y?^*b#)Wj zgxy6JdU|^1Oa-qYUjl*o@bEB&j!SOf;?dW8+Bz~Pb7gL0C69}(s(*MCOrp_fH)|Uk zop4%qBMJA<5~356ZR6v$@p*Z9G*+eC<|(PEW6B$?Y+K{9m?{6)Wn*#dQi5JfeVI>f z>hSEHVlI&G6w#Q8#(<-(HtXSX=l%>OzaIjzY+xi?V-@lSS#QR)(6sn9v&SpKE(a%P zy0HIE;d&MT;(p9kXO zX>Va+ar0f4YfVi}ZuVp%&4mj_ySuxs_b;ypUWY(_x!c=+WBzIFb2*;*hUxq_hci|i zJd9C^iNUi6z zil>EAl};knuk-T-bF#BRr)KV2qfjWx73|5yv7}cUb`Bo`=BtE;pyeHkj*sWgGt54c zVO^g(XWM#ZU6~0I^kdCpU%c@Eq^ZpFGxi)6^KPVHPPDW|fkSoyKY2>!O8QE$_UqTL zJ#vbClGx+#>x_(aFH6S{dhGlR%Wfh|gKp(mawtv}nw8FI3YO0YEPtfmd7u<!H-Br32hz&(rEM#p(poyDLmgOptrCA5GXoW!NH-5_-fG?=ZQUGy{>JlOH?A%?aNVwsVG6TR~)@=iSE`03h=&(l! z=%QsmEhucTE-o#l#w;yU8}6$IH~iHOkHJ|wM*zduqN*%jH};<7lDS^ZLtk%6Z; zt=Vq9$l4+jX8qZge(9ylOftp~tMd!YZjXH@Qp|KJhnvK!obi4cas}ck4j()7u%*ba$$p>?q0xhZQ>+5t2Ym@K1 z-xnC>xNm`Vw5ynBuAJ3ZwcVe})k-WH8XhJMS60?wZXWFnDxHCnDDJNqeD zopn>7(&ygZ-jjal34UO6cYohNg&xN}MdpH4>{3sAti(^&;$G50V#I0nR1?LSt*YFk zZYffMAU7df4^B}^B6Lqw9&yMA9PY{82|7T3tpS_T#}9Zc41K(vjG|1A7}MgC^R##V z)NSdn5tQWQ=$LD6VNokh*!q=8q|$B*Z#KMY7vd?bsqDPF*xS5w4{ub!4(X)y_TKj+ zS%}lB&VV81(ca4ddzxno-Y;J2yoMk&}dX=_Ce&)Q%LQ+_0=&$g(O5xQ& zi-4x(@`j?1^iRqWL?>Yf~7S)AP z;pJP5RO#8!`ed9aSzz6BusxTFxI>G1fBN+4A%B1Wvk=$)utzn|9zPyvY;MtT}JQT732=*7B{8D?Yh_bO-JuTiMhX@XYQOc*x$~z#D4=<}`Nd98Aqf1z%`K#JKN0=p)L;VvGx$J-45P=fs=t00Xcs|$g?n#xtkhtV z`i4k73p4XV4)koiH-oKrTG>$?r)&ws6O7#B?7I6(zIz3V6Wa>0jy9YhjwVw!uL*Js z>ZX00x=j#rle=`(?kan6Xi^S4A2~ztUY}q-KgZ5E>RnQm36bx@cNT9aq2l zZnM$qL22gm3KhT;&mywuLe@!RcGJXX&sOpWUD{TTPW2mw-u|BD#)nkn4sPpbGM-rU z5u1!PxiXt~mv@8PXGA#c0E2Qi3>E+E`2rX(iL z6?T<o+IRqGJhp$~G{Z&6a1s~@Oz8{;S<@dW)!kVV365}2tq{0h35q;wk> zH>@J(W{){)HOhqmxxlB+PA%x0yMUKZy^pnUXp4Py zbAjm4{^*WwAvZMet6{2&b916litsH+vXgsx9XKXKjP0QWH5GAt>d_U&we7*auMQnW zM5>ZQWe$PDw{M4xLM_=cOf_E@4Z1w?>_vN(3(J(1m62N_7+6=MunE-ReA|0|mrmcx znX;+8o80_X#JWa;-gKa)udn3FMi47P-GC-!!5j_mq!Lk;sYzQB@#Uvj zaogfS{W2FygJWfg*?6X+H+*^NSnjOBF&W3Kn0TU#6OGF^W{O3LEH^|~xX zQPH#JyCbhfpVq3pO!@rf%Z(JrwW$WHQ01^VQMavc&Fp8+ocX|;3q%6=hi^h0_xg!S zXpt3*@Q|vK@PZ`W_HCzrQ+pWIdk`Ii|k2XI_g%6$#4{ zw5LDIZ5-?k8?fG35!{E(sx|)-*|6MUC>HX zSqjw`ci2;r#v2ya)nU$V=23qiFe88e{+;hc8uV(D)r&1ddhycvvwD&+ zQE_>W&y4|oJK=PkAQ|+8Np(~|HAdU(1_kZ=ZuXsu>&E#)u;<1v^`o9+33oBkE5rSzugVLL--^a!!Z5wxqwq%f*M4H!0)1>2jOsKLcRziu|uq zQc}i)9&>GR-Gg!0mdYI3g;}6P<$#GkYtWz+e)%?ZuM7mbH6_gjs{3Kbcd3RPvGD}} zY^IA{S}+)F111#SY-?gNU+@2O=i(b|E9EyC(QxJs|Ho-wLkr;O?}lYW z?LMm>=4T(ej$k#cCQ`1)gERNyt&}0T8UT(?t7jm-H3LA~tnzTao^cVq^8Gb)H{&iS ze)%zy<2a1fRw8n>g#l_pjcI>$#fjmi+96fQ=y`4dwF~V<2>XdBS@MZxLWg)qtib+t zo`AgsWE7QkpcryxgOt3@M$FT!S8>x;3jDR*4x3cLJ(qIUulJ{&eon}kKQz43Xy5!$ zIY&KC67K=vl@vA(nN-8cc1UatPjG0ajmjj)9v? zfMh6IR5LTFS?w`-=(X~xb3h##eY`c(GH&pGsxG?}ZfhpcJMZOxv|deWx^lQ5MU|za zD*P%KWe{?e-tWh_^SZ}$8bpTJ}57 zt@lPK+gfa1`TRvJ<9{S#l1>``t%F4?N0nk}#E<0a?Y>h6os`e=MyLC;3_KEs-)!4#C zZizx5GZE{cr0w@6t=x%APq%s?=1+4gquBg4n6NX?sp!4Y)1u_t{G@6$+4zB^Bw|uK z!MTQ89-AAY|Lf|9-!X)pMNLppP@S|qIF}W)2>8Yj>QLgWO=6yaZd__=YL*_Om^uiu zn*bYu&r>&uaQJL`eIL3?_v9x9fd>YTi;p*W;Gw=MyfRZ_=Op$(_EBhiM7Vn^ZI%za z`Gr$a)&@o1oWhj>^0dQfu=&X4SjJ(|nWrXF=ou-e9+zUPk1jDXg>d{fYt7NP^YhAw z8p4^-pvHKy>iSgx4u>w459=A2F*7lJ4W98=w`m9Z?CtFATAzs`xPY~-O*SW z$ZQ+X^bwVAr0BD9S_xyi!cQ??%@+Wx%ne$E=N)W;1W)W~C>2b^a0gp6?8vAFvYd!R zLq7_Gc^>~rHyoka{NQ@h?;Jy8xFc`JEByc+ai|QOTX>y_ZGBB+-cGXm%F9fk^6@YX zEAP)tMbJ7lv`jMxl{Sj6+#hRh=6eNzABN*k6h|@lY;qHh7Qb1Pej4*|DRr+fyF|{n zva(9;Z0BvuZ4*fz!!Tk(SCplxN!icpOd7R7!P+=ZsWi1U1t8(Wn+Q^M;@tyKfJ>3d zEdvF=v2CqHDh-OWblL6N$W)3gKZbkNZarvTY5oFa&wlZ~+<2hAfK1yEQlTgW15f{Pm*D6 zau2=JbQku8h5y(2!7zXIZYF3UJsln05a%_v20Tf=E%2#OVBteJIi)KyIg95~x@2EE zlLuceb$B|aq>uk2ZSv=cb4AS1$BI{fz`Sm@0+9FrokzoxIvi_QSKc;TJY@%I7` z4#Mn7AEPdZ`IX2A{yI1#P~QvRiQme1|{qMW_Pry6eJ=j(g6 z09I~258j>*JVNWlsSmU8C1o-O0WPmn4jOd0_q~k~)Mvznuh!EFT z&mTs3iu}$3C=D5sTLeOt%{*|^z-5P* zEHnO?jNxX@W&hQIELE@7b81IP$;lS5gAt&VwRfU{y#yme-=Cb({pTVcP~7{+A|BBB xzZdcTuIBx>@qbs61WJ{Emn6yO$p#NjQS^0Lx5Lj>cm4h=LQ72_UIMd!_&>^Zndtxk literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/blackrock/blackrock_logo.png b/content/ko/case-studies/blackrock/blackrock_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..51e914a63b259a76b0239d93c2ce872f7722e941 GIT binary patch literal 6430 zcmc(EcT`hdvo}bWsz?zOLg-QwdXb*clps(?XE$>!3Qy4zV}Syx+nZaXv{B}k9?v>aGI)3d?6$i4 z&iV)qX(Yy8(EO4{(B0k<;Ksv~k#~1AN1{>AAPbbWt%EFhyS5PwvbB^2KNdqk5somF zjjfs|7Nz5A$96@Ax;YgVVp430N?QySXDoNDz8C zr9XrS#Q#icZ~r&7le0GJzxDk`U?*JJCoM<_<`dI$RbEKnU8}T1tznz?4Kq zMU|CR#H56TlqDpUM3i8n%0ePgVVJO(r1BqLI0orrk8*JS!)y89yej{Z_tF>a9RbO3 z6xP-iWvPP2*n|F>v$X9$=prfxlT?yY`OkR&=C%9>UH;-#ghKzLE)YN&$faffA4~rk z0^H~F_HVxh9{%=xlmp=ISir9xlwOkJ;nBsY!xeSiC%+q!X}!{8CXPqAYKh_t>C)U# zY=YC{Ba~H#m_*7&%Y}!Y-AAN!9T((3yLCX++etK(pX=O{tx@&;{d4j>I~*t&m!If! zl-4r6-FGB6?^#0bx8K;`BXKZ`k?QuET{Z5Lnj0z1%R>)>X-JK)+`f+gLV=K-CfJPd z$_r8z5KR*)A(-)n0tk1F6b}8@(B->p;eQSN_3nQl{{wYN{s#&m{~HQG{{wZ2{!irp z3(5WXp00XLzgGJnghoYGZVYCrCZiDu#1mPs{b6Atq1?)nk{B;f&xDi1!{JCqp=OFz zBgZg&%HA?INl8iX{#4n>+nqB51H}`QlQokqEm?LwuZ6H3#&gZh&FY4RQ?c)IJZ#2B zM#`O)n?Uy$W@_C`*SJ9GNtw*w32#M=70*yl1a^|a2eu8q6q&HkV1_#Ro#ufk#%onF5bFsYMlsH&2ztgDkT z7EH`p6j|T>)=svAjWMjSJG$?A@!?_C6*Lgo%pX95BOPe@&uIKLFXC|aRH@ZyYKE{lsxNUV;Gjpcw&HJ{qpbSRjn$}A^6 zQ9tW_I^5Eoscj?DUFHtTL8W#R30q^`a1u^0rd0_P6CtLqM=Qw~rK6LW#_AX0yAN`i zjqL;#Sal3ocas9;&W-yY&y0_|Q=>SudMrBx3qXLXdW?1Bwri-A5_6==Q`oINoJv$$|Y7dtVE&6Qm9jo;f}G zJ=i%qI=U7iF{>645<-R~Rxz&5fVgRyzDr9p6`CAmB{pa5x)r$X>fqp@XISY_Ph7-2 zrD@2Ld7B8=6G(OY2`O&n<45w;yH`wV{b#6KOlq7~`$R|m1@*keI|u37J7U;>-8g!* zS>iBLvz8+(f*#H1I8S0Qac5&#U2b+EHEc7Tt98S?ji_sVo=IeSur;UT3cJi>3s(?l?muFvhGbc_}uN8ot-TzE*>zF9IM34Vcqkq z`sFWVtiFWOk+rq8bzviKNi9nIp1S*Z3(RwI!P$7@3)wLjj>o^(r4P2}7~T&XYGH0H zLEj_~uQ^}L%*^B|mstDwP|HmN%d7ditgP%JOEpTv-|77PoVR155ZqOTEB<*$sjs%V z{O#LYeD0#GjJc6qA2-~#XKzzW5rW_8iI0zrh_MD9LY9_lm}87I*tG&3K-K>Q3M!9lLk$9(*i4o)UWJ z@oSRUcpYXImUEKAO0~LsxKBV}snW`w7Bs5*y!5bbjzp<_a(p}(Ot2I*Mnp}ltw*-H zTc7dqq-5I_}Boq^3@GjYOWNy75t{w2T8(Tu8e9_OEb~XL~m|wZB34B zHG^jFsjII~&6f8jOC;y1z0I@kPZjW_l2T);OW=#Qdu|!hOqSYB82UE6UrpRe+Ny`K zw$J^5fP{*}_h81YC4e@F>UOANQI5?OT_F>8%oiAIPm-*qt2f4w)LE)7z2H-v_ zcRYKJUivxTlL8XIqM#tv^Dkix2KuPEk*&!J`(!ePH)6XSs>Ds={15q=tqDp^Oe*&ZiJ^20`8KLmpS>HCKhg_Kpdb z&z~u8^z8v=v=}co*DRPcF!8-BEVjLrliBJwNS-`{^z%O6)* zxV_XFhr+ev0_?9$n4Bypn(`ldjTV`1k7^n+`JL{nK6GciaYHvfJ3}`9N>g<07f>SQ zk`mX^a(5yiro0A4$G~t`Utgc1va}Ryl_n`eLUoG~FY6j1cEX8P&E@9Jo7YxWR&d(l z7yDBfpTklv&9~#}nkjKyi0cTSG+D1v4WS7wJZi2uMQMW0x^uf&{H;y?Nwpa?$HbhtG6S0Mk&kZRjyY% zOPQ3P&rS{`baI6Kla5cfAzhSNZ$&1NOZoKX$O$eP%a#T zTdT>og|*G`#YS25Hz}5$chwOu#1^AJ<8>vx(9%d^gy9c$rg%u z2>Uk zgu30Z&CSg)tXz2ZE|4Df?#gtRE9)EpyS?TO4bw(E#JEQEur-ACB4+74qyu(g&D>t= z#I0IOXkDZZa@&0^hE8D3?-%ec`j$FqQjN_ngvRA4;zrozD!*~*T`edmcvby6CubFp zMr9&$dfMpzEw^~3p}}>0^aQv({mqpdnLzy1|Abf@$bj)1@on1~WT`}S@$vB?+c@q* zAQ0lVdmODTLDYt}{Oel9D0i zo8)x@Vh5~1aF%K>3iMiBSg4r@gEZ)WF$JWL#q*bE{4V0==9U0?YYXDB%KLgVP9}ZKp7p)U zsW9xxbE@SY#Rdh>b}WDoH=CN83dahK(PhgNKEq!`zCFV>c&&m ztUz#oT;g!YBTl$nz3cb!l7$~LwGxNFk|i)KK6@J@Yksm#!^&D(TJ^25`_G94A0@uq zm5Cij7 zl=^&skwIA-xfB}C?L=!oi({wWSE@?wnxmzmnWrW~MAP&Ts=x(>H1sxx4fOOl!M!^R zE%y`o4Yc=n+gXFkPcYT;g%)TdQgUT;p0=(FU~?|Op#sY_5u{q*tq|8J zt|q?N8HkHA1A=#AiO}hpy$<MwKLph1>vBf((J0zf_Se{p}UeRxk7W@iu*_)_(hTLpwV2A$>{6f#UN2;h&Iw7k( z%k`$&>V)7(2BEVcS{FvG=7$hQO5N&;{hhzuB2pU=5TN{Xbv`4gwXH35BT3e)!Y1)7 zNyJ)Xgjb$}1O91vw~MccC*-J;J1tnP*6+eMA_tG(*k9<=i`A}0B{f;b963Ag&c;}? z)5YA-km&pe@@KlA>`Bb;31Nd@Y`xm0U@3?dC_GQ`Jw5%^zkHR`#o&|8Lt#J4MiTjM zc+b@f{U)^^nfr>q8SYn(Jlsp>J(|Zx%JC3R2R=$a6Y2MsQ^8Ifgg+ z=q=+8L%22HGImBhB(Dp?ZgtPj`o4~ch|pHCySW6M0RhJ8*ilBlg!Vh!Vb`x;KW@A@ zKWPRO7sRnmZv>n7?(kIa;Ke}df5#5idmZ!~g~^}&*2#q8HwEEe<=l?H&nGC@(7SzC z#`U{Cki9-~S5XPfuYXzJ=raEqe~|F=#z@}G1QQdw5Abqjv?-7wQ~+5#ik9mEdG=Du zjWi;qfhRm+!X#HIDQ{Py#|nQp3UR2kN3mjrq0sqXoc8_=c8^N*cqu6j({kO&h#D2pVVXx)TJR@uw~QSoc}~%K7kcf7WZ;BPTn%bJ)fyCm7$y+#{do zqgkBtKuP1^hL(KXer@&^L#HM~Aedb$OUG4D;h%U(j{QRGQ;O!C@7uQ>8QvWo_ z^^*beXs3l5<8!t*X6K~``C^wAJ{PyTvEeDt!=tv`?W3Zi(wi)9Z*el;1Du?GXrNdw z2&?QVWoxpkCFM_B^dQgA&*zUh;F)sU92H0^D|31dUZPMnVVjf_MYe^1dPzc^kRHia zJpVov+oh!l5D+2mKR#xwit=;$DC-?}w<`V670tSs7$sz~_I z8}npkeLXjxOlg|YKYrH8Jq56T@FNm*s)WOI^wIG#@y;rZTvuIvJ;4o;pvnPih6O!p zVmQ7~u;^!bm^f`WeJEN%E@7(lEICVRVyx-I3^G(OrGx02NRn!NkBv%i%H1*Dl8iAL zVYGtoFGlyBk;kTcIPQFd8t2?faH6jnupcA$%RLSDYwj5s7*K`(FtcU-wz$~r?d=Ws zqDe9@FE8KQ#s<2KZQe|-=LMf;X>y^#q)qaSv(NSjZA-4)biOY_pFqw zgQh7_3W$a>`)|S=Z_`)LxZY%DPE;ZjHq$Llw_85FtSSt+O&{CJj@KG|^$ZOSjrTDC z3Q(`2JkbwpY2N3jN4!rKA8Ruyt}suN=f->&qqD7FneyeRYN4TOi+0?^3P&b4vQRHt!Tm2c{stPh|2C zUV9OlLqtq0On#>ee|}ebX4at0YRmfCMqW{aXB6UyeAU!zORU_wh(U1_)G%DmT&*^77!%3$v z*9U{w2UDQ?`0F&z|HAUe=gZ<587lh%T==( z!|}~U$Dt=c^D8D6Z3!{kw>T zx-W3VELSx;_q~pbp61rn~4#& z_3T@lSWQ(`IIQm)kqtSDZP?Dg5 zW!9jm%>d|PHS)5Lc%Gd4cLVXCT73T>9l`(GUZf4p1)Uilb8jm9!i}hBm;YF(D{I0F ImCSJe1$hptGynhq literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/blackrock/index.html b/content/ko/case-studies/blackrock/index.html new file mode 100644 index 0000000000..96b66334d9 --- /dev/null +++ b/content/ko/case-studies/blackrock/index.html @@ -0,0 +1,112 @@ +--- +title: BlackRock Case Study + +case_study_styles: true +cid: caseStudies +css: /css/style_blackrock.css +--- + +
+

CASE STUDY:
+
Rolling Out Kubernetes in Production in 100 Days
+

+ +
+ +
+ Company  BlackRock     Location  New York, NY     Industry  Financial Services +
+ +
+ +
+ +
+
+

Challenge

+ The world’s largest asset manager, BlackRock operates a very controlled static deployment scheme, which has allowed for scalability over the years. But in their data science division, there was a need for more dynamic access to resources. "We want to be able to give every investor access to data science, meaning Python notebooks, or even something much more advanced, like a MapReduce engine based on Spark," says Michael Francis, a Managing Director in BlackRock’s Product Group, which runs the company’s investment management platform. "Managing complex Python installations on users’ desktops is really hard because everyone ends up with slightly different environments. We have existing environments that do these things, but we needed to make it real, expansive and scalable. Being able to spin that up on demand, tear it down, make that much more dynamic, became a critical thought process for us. It’s not so much that we had to solve our main core production problem, it’s how do we extend that? How do we evolve?" +
+ +
+

Solution

+ Drawing from what they learned during a pilot done last year using Docker environments, Francis put together a cross-sectional team of 20 to build an investor research web app using Kubernetes with the goal of getting it into production within one quarter. +

+

Impact

+ "Our goal was: How do you give people tools rapidly without having to install them on their desktop?" says Francis. And the team hit the goal within 100 days. Francis is pleased with the results and says, "We’re going to use this infrastructure for lots of other application workloads as time goes on. It’s not just data science; it’s this style of application that needs the dynamism. But I think we’re 6-12 months away from making a [large scale] decision. We need to gain experience of running the system in production, we need to understand failure modes and how best to manage operational issues. What’s interesting is that just having this technology there is changing the way our developers are starting to think about their future development." + +
+
+ +
+ +
+
+ "My message to other enterprises like us is you can actually integrate Kubernetes into an existing, well-orchestrated machinery. You don’t have to throw out everything you do. And using Kubernetes made a complex problem significantly easier."

- Michael Francis, Managing Director, BlackRock +
+
+ +
+ +
+ One of the management objectives for BlackRock’s Product Group employees in 2017 was to "build cool stuff." Led by Managing Director Michael Francis, a cross-sectional group of 20 did just that: They rolled out a full production Kubernetes environment and released a new investor research web app on it. In 100 days.

+ For a company that’s the world’s largest asset manager, "just equipment procurement can take 100 days sometimes, let alone from inception to delivery," says Karl Wieman, a Senior System Administrator. "It was an aggressive schedule. But it moved the dial." + In fact, the project achieved two goals: It solved a business problem (creating the needed web app) as well as provided real-world, in-production experience with Kubernetes, a cloud-native technology that the company was eager to explore. "It’s not so much that we had to solve our main core production problem, it’s how do we extend that? How do we evolve?" says Francis. The ultimate success of this project, beyond delivering the app, lies in the fact that "we’ve managed to integrate a radically new thought process into a controlled infrastructure that we didn’t want to change."

+ After all, in its three decades of existence, BlackRock has "a very well-established environment for managing our compute resources," says Francis. "We manage large cluster processes on machines, so we do a lot of orchestration and management for our main production processes in a way that’s very cloudish in concept. We’re able to manage them in a very controlled, static deployment scheme, and that has given us a huge amount of scalability."

+ Though that works well for the core production, the company has found that some data science workloads require more dynamic access to resources. "It’s a very bursty process," says Francis, who is head of data for the company’s Aladdin investment management platform division.

+ Aladdin, which connects the people, information and technology needed for money management in real time, is used internally and is also sold as a platform to other asset managers and insurance companies. "We want to be able to give every investor access to data science, meaning Python notebooks, or even something much more advanced, like a MapReduce engine based on Spark," says Francis. But "managing complex Python installations on users’ desktops is really hard because everyone ends up with slightly different environments. Docker allows us to flatten that environment." +
+
+ +
+
+ "We manage large cluster processes on machines, so we do a lot of orchestration and management for our main production processes in a way that’s very cloudish in concept. We’re able to manage them in a very controlled, static deployment scheme, and that has given us a huge amount of scalability." +
+
+ +
+
+ Still, challenges remain. "If you have a shared cluster, you get this storming herd problem where everyone wants to do the same thing at the same time," says Francis. "You could put limits on it, but you’d have to build an infrastructure to define limits for our processes, and the Python notebooks weren’t really designed for that. We have existing environments that do these things, but we needed to make it real, expansive, and scalable. Being able to spin that up on demand, tear it down, and make that much more dynamic, became a critical thought process for us."

+ Made up of managers from technology, infrastructure, production operations, development and information security, Francis’s team was able to look at the problem holistically and come up with a solution that made sense for BlackRock. "Our initial straw man was that we were going to build everything using Ansible and run it all using some completely different distributed environment," says Francis. "That would have been absolutely the wrong thing to do. Had we gone off on our own as the dev team and developed this solution, it would have been a very different product. And it would have been very expensive. We would not have gone down the route of running under our existing orchestration system. Because we don’t understand it. These guys [in operations and infrastructure] understand it. Having the multidisciplinary team allowed us to get to the right solutions and that actually meant we didn’t build anywhere near the amount we thought we were going to end up building."

+ In search of a solution in which they could manage usage on a user-by-user level, Francis’s team gravitated to Red Hat’s OpenShift Kubernetes offering. The company had already experimented with other cloud-native environments, but the team liked that Kubernetes was open source, and "we felt the winds were blowing in the direction of Kubernetes long term," says Francis. "Typically we make technology choices that we believe are going to be here in 5-10 years’ time, in some form. And right now, in this space, Kubernetes feels like the one that’s going to be there." Adds Uri Morris, Vice President of Production Operations: "When you see that the non-Google committers to Kubernetes overtook the Google committers, that’s an indicator of the momentum."

+ Once that decision was made, the major challenge was figuring out how to make Kubernetes work within BlackRock’s existing framework. "It’s about understanding how we can operate, manage and support a platform like this, in addition to tacking it onto our existing technology platform," says Project Manager Michael Maskallis. "All the controls we have in place, the change management process, the software development lifecycle, onboarding processes we go through—how can we do all these things?"

+ The first (anticipated) speed bump was working around issues behind BlackRock’s corporate firewalls. "One of our challenges is there are no firewalls in most open source software," says Francis. "So almost all install scripts fail in some bizarre way, and pulling down packages doesn’t necessarily work." The team ran into these types of problems using Minikube and did a few small pushes back to the open source project. + + +
+
+ +
+
+ "Typically we make technology choices that we believe are going to be here in 5-10 years’ time, in some form. And right now, in this space, Kubernetes feels like the one that’s going to be there." +
+
+ +
+
+ There were also questions about service discovery. "You can think of Aladdin as a cloud of services with APIs between them that allows us to build applications rapidly," says Francis. "It’s all on a proprietary message bus, which gives us all sorts of advantages but at the same time, how does that play in a third party [platform]?"

+ Another issue they had to navigate was that in BlackRock’s existing system, the messaging protocol has different instances in the different development, test and production environments. While Kubernetes enables a more DevOps-style model, it didn’t make sense for BlackRock. "I think what we are very proud of is that the ability for us to push into production is still incredibly rapid in this [new] infrastructure, but we have the control points in place, and we didn’t have to disrupt everything," says Francis. "A lot of the cost of this development was thinking how best to leverage our internal tools. So it was less costly than we actually thought it was going to be."

+ The project leveraged tools associated with the messaging bus, for example. "The way that the Kubernetes cluster will talk to our internal messaging platform is through a gateway program, and this gateway program already has built-in checks and throttles," says Morris. "We can use them to control and potentially throttle the requests coming in from Kubernetes’s very elastic infrastructure to the production infrastructure. We’ll continue to go in that direction. It enables us to scale as we need to from the operational perspective."

+ The solution also had to be complementary with BlackRock’s centralized operational support team structure. "The core infrastructure components of Kubernetes are hooked into our existing orchestration framework, which means that anyone in our support team has both control and visibility to the cluster using the existing operational tools," Morris explains. "That means that I don’t need to hire more people."

+ With those points established, the team created a procedure for the project: "We rolled this out first to a development environment, then moved on to a testing environment and then eventually to two production environments, in that sequential order," says Maskallis. "That drove a lot of our learning curve. We have all these moving parts, the software components on the infrastructure side, the software components with Kubernetes directly, the interconnectivity with the rest of the environment that we operate here at BlackRock, and how we connect all these pieces. If we came across issues, we fixed them, and then moved on to the different environments to replicate that until we eventually ended up in our production environment where this particular cluster is supposed to live."

+ The team had weekly one-hour working sessions with all the members (who are located around the world) participating, and smaller breakout or deep-dive meetings focusing on specific technical details. Possible solutions would be reported back to the group and debated the following week. "I think what made it a successful experiment was people had to work to learn, and they shared their experiences with others," says Vice President and Software Developer Fouad Semaan. Then, Francis says, "We gave our engineers the space to do what they’re good at. This hasn’t been top-down." + + +
+
+ +
+
+ "The core infrastructure components of Kubernetes are hooked into our existing orchestration framework, which means that anyone in our support team has both control and visibility to the cluster using the existing operational tools. That means that I don’t need to hire more people." + +
+
+ +
+
+ They were led by one key axiom: To stay focused and avoid scope creep. This meant that they wouldn’t use features that weren’t in the core of Kubernetes and Docker. But if there was a real need, they’d build the features themselves. Luckily, Francis says, "Because of the rapidity of the development, a lot of things we thought we would have to build ourselves have been rolled into the core product. [The package manager Helm is one example]. People have similar problems."

+ By the end of the 100 days, the app was up and running for internal BlackRock users. The initial capacity of 30 users was hit within hours, and quickly increased to 150. "People were immediately all over it," says Francis. In the next phase of this project, they are planning to scale up the cluster to have more capacity.

+ Even more importantly, they now have in-production experience with Kubernetes that they can continue to build on—and a complete framework for rolling out new applications. "We’re going to use this infrastructure for lots of other application workloads as time goes on. It’s not just data science; it’s this style of application that needs the dynamism," says Francis. "Is it the right place to move our core production processes onto? It might be. We’re not at a point where we can say yes or no, but we felt that having real production experience with something like Kubernetes at some form and scale would allow us to understand that. I think we’re 6-12 months away from making a [large scale] decision. We need to gain experience of running the system in production, we need to understand failure modes and how best to manage operational issues."

+ For other big companies considering a project like this, Francis says commitment and dedication are key: "We got the signoff from [senior management] from day one, with the commitment that we were able to get the right people. If I had to isolate what makes something complex like this succeed, I would say senior hands-on people who can actually drive it make a huge difference." With that in place, he adds, "My message to other enterprises like us is you can actually integrate Kubernetes into an existing, well-orchestrated machinery. You don’t have to throw out everything you do. And using Kubernetes made a complex problem significantly easier." + +
+
diff --git a/content/ko/case-studies/box/box_featured.png b/content/ko/case-studies/box/box_featured.png new file mode 100644 index 0000000000000000000000000000000000000000..fc6dec602af171e0cc744837848cafec610ce686 GIT binary patch literal 6348 zcmdUUS2P@M(7sL*LPYN+I;*W92+@1*CCkd{ZMD@m1W|(My+>#DvU>N^dkfK7A$sqB z|BLVbyZtWScP?h$nRDjMne#l)8?LFYK=6|4B^nwUfs&%E_Ol&)J}G!O&$l|sr~w)p zMZA*iCta_MgG}!PQv-jLmrmi$MsX1n{G>YRHO!A5MIoc~|F8$ng z<@D%P*<5DA(>5nysd90b!JNFTSK6%S(Q(wcs^16JV#e!M;{Uvljn}OJiOY5yIk^Dgr;Z_uS^|MrklR57rGz0GNz25K8Nx25^GL{Q&aM^iVpl5t&e*`lQlGu6!bTP^ zms2(mP0zED6C13%!iX37TmpT9?yMrTpHR6jg?N2NSBYOM(}s^k&pn)IEOd`ZNmK+h zQcDa|d47s0Xcmy-eKHF)))flk%Pu8)v1amyw<(iggQbln_&JqUX`Z@vAelWAci>)_ zlDnL$HX;!#5$60xX2T~fHBMo27Job;g~OXddEF;1G+54Yp;S1jCzkkUZ;pEpYGO+nUO`s(mrk{R*hN#?Ehd(ewjnB%tLL zc~+m>^k(1gojH113wMSX<&-Z9LNJ0Y+p_S}b(KAhyPGlc=kYn)3&9C2D+o8i`SbR@ z#?^Myez>Ko8p%Umeu?IrZ4;UvURh3^KJPEXSd<##uFwnghs8<$>6g zYH9Z0ztoLGQ1t0YJ2%U?4CPk=tv3WiDZ?D>m<9E#dzBAYg5tm)OBq4&YFI+H*<2@& zNgA@4KQQ_#<+*#?kTKGLNe-dI>X=yWB;g&))sYg+v zOOqYbbYQcRP7=U`%wea-8_u!*)oFd`pBS5qp+0zKvOCNawD=?Cl=mN7p{OeS%m=YE zo5j&SCpkJ6rFOs4U^Vx?SED&NHEuvsQnKGH=0Yo_p-aI3gz{RfRlsv9@?JGi0AEVS z)s)WP>Svn*>&*kK{v@m3;~md8e;MzLywEWItD`&|uQW0E)M#3vi1VQ$FGnG#NLq)# z-Oo(lP%RD+0|cFG+T6AK^AH<+#dZBd9BFaLQ$Csuzu-2Xmmki+Y;hR!x~RGe6*)vk z{`7rFx41ZP-;U1T=Vzt(#Bl7t4eFvboA#4yXuJQaRs=RVsi1$E5Eq6}V+q$}7nqY8 zsKq}0wRrG9#jMJQs`QpD+OA6=J+dy)Ye+iTbd}1kQ-?^XIuZbCJ7Rx`U#|sHdA0hc z!To;`G6kk^%@xtc%+i)jphUAvUdC#sVgLqM=mJ<)fEeV;KEv zx<8>G)*)b=-vVAI1<>|KOx2IbK z&Oj#hS1mU<$xG7ODzoIHhb{Iiy8mhl8HYvU2wR{zIJ=aS<;?Mqm}PAmq=>%x~)rB70C%zWMMOf zhtex9ux6Ez<@Edfe7T4l;wC!7K-&RvjZV&NZ-pj02?~iOspOr{8aW#?>1ROwN=)<{KHd%X}s#%egTRY#!%3 z=&jT?aXz*~DaYM4s_&4DiUwcm@EM$3ju4g(4jRyhTOYDvcI!w0Onhp*+H$%$SadZl z$+XJsDLF@n&SQL3|fK>f>q_tJp}%e{(Mad&tTs$aW8jBE=hD*Dra-6 z2(X72_f-6TgMBd;gVXLhz{mV&LJ5P@N$1-E`4fXWax=rSs7TRfm zlSO)(jn&8f0>5nu+T{s}zO7)1#z_f~rq}U>Z%mCi?e-{_&e_MpxS|`B1he?fGEst{ zHAKr-YHWRvXBfF2F|C4@5mOAV_UgKVv~E{|-X+(g8h=UkC7OP#s0OpnJKL@c11gKk z0HGccyd7&^;*uggtCAAEhvfZ!u?hApzgO!&98FB6ZMhvCY(4#>w76Qa&>bmtsl4|g zXi%{>AZTobh?*=-&#;xpQ+=qg;4|15;I+PaJVclO{<64fy~D~|z8{m<`mC$5(QT>z zKzvc64}Q0T?%pfaM^Gn*3f#+(nJ>j(ZG-ZRDuM8`LjU*+-aoF=cgo$72S$u5qs)?o z*=|;xL9z;JTlnpF-Z=XtV357Z^QB6=Z^rVWlasW^Yh>D^?&8xcg>L@R)25c&XbOtA=E?~#5(lfHi@7tMs2YN`lG?j-*jL9u|H+>%8njo#6Q48Pc? zo)8^?B@a>HaN3iYtjZ zD}goC1{iJpSe?IxKtFnnDyvRQEoEg_ze3#0&#aMuXY&eR7^n#zOkSOr6Q`9yam|Iv zo4j=F6sFrl4HI6T&g3fnXc{yHR{Ck7XdBv~l)}zVOip^Eb0x21ZGwW#A#Th$j>$Wu zN=~hLfxSine=BSw(;FLz!i^rSGX2-r&}$O|?zxafH@kYniwiYPQZvMNCJ8}}?uFiW zg{HXVLGc$X&AMz`6r1W+Lv`wdsG;2rlZrjFqoDGp6hRYAP<9ixC3IE;ahE6Rek?*Z zs7AG8``vDwJbJ@-+f={HX7#xlgVmeCSw`aeH8M@QcT-v`zm0m3r!?Ac`sQXS`4T`R z6(Pz}mF4Gi3C)vHK)s ztQh;bxWUG+cEifLdh>4Ql&@9?<8A)4--*+qy=;kd_iiDm1x0&n>Kw#Ji9M*YKY* zK3$v9uELjdoI0QG@EQAH!vJ#E5TWq1REZl0t%lf^M2J|m>rF-vL_`BYO@&cuf;JT$ zn#B>Y&vcWSw6+E#mky*EZoZ|hn|p5w+sYig1%d!x8oVx!W3M&rPg!`2@Ant$c)ouA zf}w-Hq;D1Wi+lAnA+mH^Ovv2VbSf^d_Hen^0mbzU{1tWik0x*ll}Bi>YdHg1PWS4Z>UcucADySA=}(yX-UF3~=~&r_G+q+y-;?<2OC-x~)_uST zhxXp`!HH}5mAHUXWayS@oRodjk$Do_yLi4)Jdw2mT)_$cnO=o_D3Ul=Gg}a3F3RJS zb(OOpB9!9~GIv)T+{17(`mxEzvvNaC&T zfcs>X6(!2&+bS#As*bR{xQyAl*@>pB(|^>|hg?<)gk4GXsp0sz`q?V*ZepV{Dz5yV zE7cS*f3rw&bAnBL-&z0=wu=XJ2NGEixr|w$*25V9B9lQ0&WW}Y)9Lvr8nts<^CPLi zbxS02h5uMs6esh7UmvSJXT7!O$tx$hWRQu|b_^cpYE~rR7$x1OnzbZ6BzZ_#Kp_Bk zWz;5R@|LmuVJLm`=tHR!2szzS*`yiGdT?%;>Wr2#y7h4l`1Cr6kyE|+{H<0KcCznh z3P_>*WxCUlC-N@!lO&k{{*%x*?ks_|jU5rFibYGB^AXinXBQNHDPbebQAs)=w}f~S z)kBEx!9?Is)(`*o(mz@!UHLt;u&K0Y*X=!Hc&T&4P`TEdZ-2{2VV(qhKsQfBZ!9#u zC7bG$YkT{ZdzF&4H6Le<{qt6x5bP0OkwES)f>ioWfRewnxBD%MX}b|=XFvq_j+J{|LXW&~CE*HK(l$kS8EZct0kk?N=At#DN6 zO=Fv7!k8BKl}{08C;~075bSN1LCYJxk=Tl~6~mj#amn$GKz)BTW6ksh%*wCB^TAJ# z9f`k1Lnk6WcyM(!wItbQlk{IYXEoG}cJ+RC)|~48bfQTz{WJ6(VPT#jclBAKQek2q z2Q8x=CpZ$AU`r6+F#NcV0XwAH0tL*)Y#YW|xKA+W5#4f=tc{{dRn3eBRDYDk<$bk;0vqxRL`{!`stoMc zs5&@(Ka2SEXrVyZ`z_yZ$Y6DiM#t<_@fTD1{z#R-8IRco6fPiK?5#>hdr7c!bF==! zr`b)!>#_gp=#pQG+?1(sV=NtDW#XHb^&&m}=;MFKC#~>2rnJ+PN_wC@KEXv4e*>Zd zVq#4YrSz%*m{oLA2X^RRM=7v+-%emHo?sXbNy6>lzujK-5l!cM4?Gs8H|rVh!Txvx zyMqpI_K_}1n7J6v8V%FZ?}7gW9ey_G(RgizJjgO&7xl@~)Lc{hT~}XpQ&k^ZwkkGiTz!B zrH{kg2D3vMF%tx0%bnB#aDCAByAqyXw;dgLT1T_A>|!K>IFF%a!ovGiA8!jUT1z}m zQj?xB^w=R|yBj>l*_$gMvf5L~r>3{k^iO}blXH5#G&vLbp_tfet?fnPxgD=r0Py;_dkE}ng)2p{^tO`|ryhc&dkH%+1UFt#=MWqbf$oT%V`b(J{ zT1Wx9o8U|;;%;wq`#Bj%sq5D+T8WI`4Ed%2VwF%`E%f~F!2>|`*_#-vFtQdAQ295sq#)NxDWhw zN?K17{MP-7jD}n0~n~`+QyV@<>W3(NQI4cWrUj|f+=vKrcN^KN4$VJ*RM(a(|k=|}@Hb=&e zBWrF;V#W5V*W`(=J}rLSv5@16f!n0h>g-0{0!6guJ+R+Jv$~U)0d=)3mYK^W0yXYx zV3jkZ8&N#9*A&1@#u5V=eet-}Q-<6wwc1m6wP}E`fAoUVKHV|fFS=++B-BYxYmJfU zy;jYvf^(HCXyQ%^0zxmn1`g3k zUF$a?eY2~@KqlCB`+0W8f%NqTi?%{3UyaSHA*%ffkR8cA< zr|Oz?#l>)T!X7?FSsnq($i|9_?a@q1gpCs5uK|Gj-H{g!70;>eIFPVC17cn-mAkek z(zTGpx;t=++F{oErn;P0z%ho)>$p~W3FNZRXRGU8LhQ73P;B@TUaqAD{3H2v_Ls1H zBXpRWTw)IRV{@9Ny)Hkw2t;%7mN1JoKH@kEWHPMhI5-XOKErx5h=Jqw+CToiZ}fOo zt#4dldDMZ8DS1NfU@voPF2~qhG|7JY+w#)L*pIk6zAOgQ!g?EPHd&za9j|r6Xore6Jg zpIQ@M)>c{veG%_2s{Fg=zi&6L=#F~z{{`<{J51OoL(c#E+V}rjv=4s5YJMByP~f>P R_xwwNrX;5>i;y-C`hRemAC>?B literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/box/box_logo.png b/content/ko/case-studies/box/box_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..b401dec6248c6b4c83e131158e57c7965fcda99f GIT binary patch literal 5988 zcmd6r=RX`yw8lkQ5riaalxWd=tX`r;L??RhWr?zS%};bfqOC3>I;*$nt9K$SQFqy0 zBCOTB>;47z?R_ybFXp_M&zw1Dp645{tF1=<==mc&JUnvsw<`MgEBt;$JOtd&p`ym( zczCpY>MDxw1M~LsgT64HWMC9}CaSAC-i4LVuew!q`1z2Ma!?L20=zHrLN`c4!Zd9t zCZKYNx_+4nI|2~HD;awkx>_wVCx7~s&s{R7f&Bn`K~nR@kA$KPly6ylPf9+Vtj@3c zIXd2+rd0nO<86Me=FGivBL9Tiu+qr!w({@Uh8t93>XXG9fio!TgbB!dS&3 zl9G~5Q1RizClD6Gs5lKue5HYx4rOalxz1F zIH%;;bZj0m_Rg1muz+0`xd}jAD@_Yq(_vXRI+0!$?`zud zrrjnRI-mBxoFFpf6D-O0Ch{gt0e*Z#1`(#c-yYLdyNPAV6J`_SJ|bwAjD|7lHP0$p zis|0Q72&_k)W~8V@KL#d77Wao4Fu;sn12d^EsF0s%&1PiaTI^I9#S$BQfm-{;ecP1z@=QcJJZ|Q)Vj8vq?k?lwb1WwgC*DKQQG7bGtSjbQ>(wZ=rzox zlyK(Bt;>evOl~E|I|A#=^nj0=3yAugG})GjRKDeBE5UtiP>b%5m)px7R#Iw5c33Kf z>p5YCKo3S=#Kyaz^>lp{Ex`fwDN$9&kylRsC;Hp7s^aT;sn09^wy(;7Z-mp8K$GZC z>dY%ZeB1Gm7c@2|E3M6ijb`@OWdZyH(#d>^qL{<8o?TS)t!p_st;%D`9M9`+>jA$? zVz0`X4DiSU{y~j?9{D2|+T|GiFW^4WvD3b>NESggfMUW+{$zAElw^Ki*8@jRDY zpgJ&-Z6dwq0>{U`($$`V809x{YX8EA_%=bkrO#*h!n?$&<2(5pZ3 zJO7+$%LVHM8rjsE=sf+9e7azn0VCz9qv~0d8-q)n2&bM-D^26e9w1rH-1ko^ziXk| zBW~TpEG5IOKJ5iZfdA_>myVd#%9m=5ELWAs9!3vS9FFX6R=X#8!Am@%?bA`AiofkZ zSg|utdKOIQiIX*4CAmZVBnT6CUi#(17Nc&0kmbr{=5RTnazP+egBO!o<-9U9*i7Rr z_Qo|iJE|XGU<&Ex$z3frM!YgjlP*f3J+*EctQax_>>4vrxL@ ztA-3RZ`n*4T%)>pJYm1!y;C^Rc;uC^&YZV_u(V$bOf+0jEqX(yKPO&zn<8;B={$ul zsLDKy{glR-H=#7Z|5?Q0^<(*)ZACL=ReM`k=#3+2kI1}*Txo(gz?g|8A5$yd1{syom! zi^8JFKhvjGUXe%+O_^%XMDJ-}zHFy`R%`&7;()ZB2HrBN!I7l<7&_=~qOTD)A?@}} z5k!~Q5_CQ3ee1f5*8zA%6fhZJMXrfDuZ0Mqs*D8?7yGWzU>68id7R#UOzELTOXvfI z+Yw$fx9V)L>^cOQ5V5>x>Jy=>;yTh+?U9T-N-5M1K@OZwpc&Bl0c#iQIwvgJMC)O$ zU*g?uZ(fppe1FLd3(Xa7cI3|FfLXM0XnQPfEfw%!6ZUC;IC{BA zPxp5W1v(1vwST#Fey3MY$GMsAao-MFlB9_A?wiKdiJ)6Ak*`+E6-2BoY-#@i#D-jSMJjk=Zh6~;vI(3 z*Ax7fU&vU9xhEBRBBXZzxV=oQetiNEXu{Z1_wJ)>JoAt;SzBrzizqLP z?l3n`+4ra`2DW#v_Lj|YCTQQ zPxK$we~&Ze2A|^!ydN@@+*Bh&Mi)bU_LM%Dw#IF%{|K2>t2DB3+|jummWn;f{C!a0 ztTmv-C1Gl|$$ERFT4REMan!EV>ZcQ;@uBD=1~bW8puaxG@R8t-<^dcd6xrE7kE}$@hrLYgJKlOcz#m-M~(Q8 zwxCI+@6AO%vs8;RskYy3YCcFe{x{tNu5VRCZ688SD(iSW$`3K%=bCArR_mjc!LE05 z4EOCLTetJ>nb$X|Cy3Okfn|VDa>k!JWCDuRNk7d=l5H1;`swIxh6K~PnlI5I8d?I? zg;m8E>85?!W=t3L4@xwa+ky@IqECnevQYKKtk}a8hUNAThO6FOoW`X0YR}NA{!t23 z>}iy$^XiIayjF=MNH3j8?`XRw6!?v7p?OF>N*mfTrt{g+3;jLIIV@pe97LrA|1qgj zN;TqA(R_Y6&lFW`G$=iZ(WhC(K*@N$>L`n7$zMijF@i8r?ZlD`=M* zmol1Z0%4u7EHuQOU(?@&XaJ~_TKO$u2XSi-bzWUWi)VLiM^^2!yFv4Vb>hWVHYlF6StprCsR%H9}qTxgQ+K zv3L5q1TXkMREn>pHrDWd)l~gsz%lR-r%Y^&28d1@ML)K-vdCkEAl-Mm=}A@NelE-IzgT zb|p2O$OHGwfiL_~OkV+x-;=BaGx2e&+3D)_7%BK_Qf-N|1-~ZfvR86cGd#06kB)su zR){%=zb#-;R(bk`Rp!Il?Pu_P1liqQ{Jv5c{41y_RrtTS(bEE|0NK?^lTy1Q)2t%5 zK|j5a$Sl{)1SB3(YW;Up(AuCE8nh$3E&9hoX> z&eyp&7DnQ4T;l1w%ju%|{S&Rb<=?;uMmuE9`Zqnb4=k`v0sqEMNYlF?Xedr0%4F2u zE^{fEa)AXEt~b9yifrk--6OEK)v9`InJ|Cgy@c3+dnN}jU1U7T=rrJk9-E75{6}Ja z6{cVwT!!|kkn}7!^KKqew&-Y>8aESyPBfFUs=@;^udL~8xcz+aWrPb*w>7a{n>3i3 zb@G}OP^0E8riHa&uK*?&@1kf03U3cdR}1*8*tBs;9i0)YVGas9J0vvge??q+K7@T7 zl#P1@wYT+>iao1w+vwhvG0lIdyubsuT675hgPtl2krWn`d&OH}<}L7$A@q2Kv#!aw z71hf9B<;$Y8Ch8H!seO~PIWchaJ`o%_fIx2203(~DGrGKb)DeVTGBfkio*%E9G7Zy zKgYOC&RwOhTviEs?rl+!xb^zABa(u1Hd%kBN_8KMJm@`14ZT^MVWa?K=l51xy5py< zf18z$ETCwkvqm$a>bJNhGN4KHtVyw#b zZnno+9lw5>HQKXG8jCX_XXQ4dAKBj#!fm*xJ|utjrB1ed=zC5mE}el1SXIxQkKQRy zWCnAsj^>RQhj6x`W(7rC%qvApg$b7({p_Pcp!+Ni+}(!`6x07Slw{vC+l+#|EcQFP z(>{-Py|0_s0y5@};LO(3xdyWDYG3oToN3K2Q95dAxKFi;-da?+c}HsvA}_h`eu4*< z<9K~R_)bc*J=h#%uU8COB~E3L!-5-I%aMB~y(Ql)G)h~rjcD(#(1%qi__>Us(Q^#v zkX)m>8GYHtfa>?vCbF#+bUMqj4GQX-iXd6dMKPzzwVCIo?y(XvjiJutwjb^+h1xsK zD?bXZp@3V5p;vO_?JlkY3M=xkuIbaD2V@bw#k%xf;}?^m{>r5FN`u}5@AH-zH%D*O zhsu0P-fh9qGLL29`CG_j!u-?`Qr#}cTUB@Cqb`Bi_M7+**mo}uIp0-?ntZBqttRny zbj~{~Fq%HRjB72Bpnm?=$3#VpT|uMXPsx~bB512FGi+M?K-;QG{Z94 z_R1l7g%L=y98hJ8+&Gz`kl@GJrhX3Cjz*tS_ zNT;`sKRYp{?#P;U)IEGup|5Ml>Gj?Amu;AufWpnTtII~1!s6h*2*da8HG04ozxZf! z@h~L+LhGbewQENXwnuD$ws4ylh{(Rk6XWpc_A+G%@N_RL9Rqb%iho0ti8%SdGc+z1 z|8Uldt<9q3@W`DmDZbnA@A=rsjzjyXx<>ia=!bJ12Jbel4``z^z!!fyAcwS|=gxnd zsp^VqVy>WjKRDeZBM6o-Y{6-CK0(_$=sPe0mE+sPmoo{%|}H*Qqr3%CNC9t!7gdJ4KBgr=>JTW*lMY=D3d?#?J!7M%Ipd1f9AV^1Gy4_+4u z{d>gl1d=lKQ~1uylGwV-j)atr#vu=ulgQ2&%ZI&5YhH|(!1!ouBo&6*5Vcy(Xx`|t zCEq{Y+qvC~*;pf=&7IF{3m(4Ihx-NZ%5=bf#Zt=*IjmPz7DqSg(E?53M&mD; zyi;8vpGNtUtDHw&jJv-nDiS^}n1Arsj;1wldBF1CKqR0@*wE%WX39K&XrqFhLn1TV zRM#Dnzq&4u%kY0_o^%K-Pj%#%x29sNg9;2u)b4D|?dF(Ty89MW6_pR_STnpVoPvbg zux&%rVYYf-hW2#a8U#RAMiQZv*L(%mI7G(-2$UD7#pBN9^5DM(3&gn)o_gMg%TUOaX0 zk8|!`Yws`C`#kUS#`^aDv7^;h<*)(d00aaCYz28~&F9hT`4&b;eSW5nO#FEs$lPTh z?r&jM?p{z=O9TlEn3*Mwf+N)0QqvM@;o~xFDU5)C#A2%jafhfV3z@?lIiY`TIK3U6 zpV7}`V&2YBb9+m78Z%34TPG2^bTt1kakm$t`==;~iaL!H%+->H zj}y#c&c!1@!!O9m#V07h&C5>14dMa=L4rUoUJekK5J*r6#6|P3i|*N*tA&-2rnKz8 zd_Av3=xp5GorQovFE1}nFCI>qt2K~IP*4yE;s$bab39vcxcNA_L%lhi-01(!AZ_Vp z?rQ7oZVPjw`O643gL$}%&^;^tX9$kY|Dknq`&XKt69)8#Is>^lL4QN~x1oy4|KHTn z@jupX?wXeWSMPr%cGL24wghTgy1_hL&7TWrMgP~8vyhalCDa|}ss)2N{5y;4HZXUX zn+?pFMoQ{$*7#{yRG{XzPJcIF{{y9>BBbEt<_>i-w^Wc8p?l`xw6(Pm5|jahWhG<< z1SR++xwvHb1!Q?7r1%8+r6l;dd1QHI{*9G}nR_@|I=TNFYw?WbkpzLHxuw8@5@4?X ziv8OVj?T}Xr7c};JuNL{U15$i{|H>j_J3=6=KEh-{)x5tFIxT;2mk-X0-wbI|Caav zDfhp(o_px;^gqq}y!cQ1TRJ`Wyz6sw>on&wA|OyTDM(9bc`y7n#PHU-XnI<;+*z5Q zQ@#{Q6sOA>Dny46>*Dr3%oI22818G%sW~bgnsJQuHjD1-G$~fyriK)jMFEEBOcQGv zp%{A1S)&%0+v6QP$GLCF$e2=au>x;9fS;+DN04ax``@47_A=|)FjKbA`fqOx>ygv?oQEoq_@G6aGZ7TUdea%v>_^i}{lm?Kt4ftOC| z9(NgMapdJ!Gj;S|i&Y3vaz+vM)K@X4$Xu#%tSDUKG%>9WP{d25U~VjT(6R^L^*9dPE4V3^q14ja7&=lT=%*_@#!38x&8Y z;nqR{>jZ-Xeud6yln{Mkr}?=JgQGJ{H}O`oj+loZysQVtMKO*F7Nd7Z3YrLeWbn+~ zypW`D0owx{(W7&~T+izPZ+7Edfb=Uw-P)T*bW|HIbM?Bu5{*PAYn!ER@RM1oSUn3V zan-4DT2EYKu2LuolN8j+rxvrw@aV+w`bx?!ZNoV|&o98e`tlf6b#ezo4lS)-p(j>i zY=epTN0U#`$_~JY3SCC-Ba;>$C!(gAgYZ39K}cb_TW;RTp^DLq(y^~y=IU&DGj1_P zWlcj&P5Gv+s2l+Orn*iKDWynQTp|DI{t<>z9Vff0p;n4^O2uvnP7pH9ZHxCWsAv@ig}D65<0+?AQ0ws(>^-#p*5e$??}Y_WHQ1gLP78 z)ytwf23xyf&-7xUBlntpli}a|*&egf8kxIYIXx7=SFw~1tqqk;GSEdTGV($KFoT)f=K|imU`WLuWEE~38n^kwWobZ5tyC?uu6p*- z2}C+`O1_;qptAGjd9@Et^ZQHUD0b+ut-64dPz_EF#~ ztu9QRL0Q;G$n!cUSL3h=u_*0;0c!9u2IQqTz;LB5M`%na))s>^m zmGq)CJ}1LhG%uF%l3EvZ_iiR0W|r2xRLBJ8P&-GjXmP5|Gs7KkyZdrnR&m^kMGFcT zFXK(hm#@y&@rJ47c@BEfk><7_SjAy}&eU2Q0h}kK?(SJx=V;H!fVGL*6ROz1PgJmc z2TmWf6w$GR0hzhBp;|&XlxA8tKkwp?>7CH%xav3l?C=$srpk5qn+1f^0l0h|I&>zr zl1mLln@4#ktab*8tja&Ra@_xNDK43(kHvb~yVSf9few9Lv|TGtBOYyhef}%LjNrKA zpy!NDGOz&nfD$>_D}2Q-1bo~bhx?s2;IdF!i6-W6QFzz;)|lVNexQ6{dpRp#nnV-! zJ(yW?Kv!?2ZF8x|5zylVm~@SnPJNq)o-b#(p>(^3<71MZWYmA&Og%m6^#{WMQ!_4rF}!6|@|LkhBg zcDfMEto(^>^h5joX7+rSq4v{R$?|*tGHm7^Ye`xLz3xYEkBfN%z=c7M5iQvWP1#P+ zt({RvFCKf3Pw8D+;>$~!B8Q;eSqvf>S?wXbs6>B}AYXP=W@02o;ukwFh|dx#n0M7! zLFFsGE38pSo#lfJB%v=}r{~b3#^ff|JRMwElgw^GW(+jSvX0_2+daMJ++fGQz46NJ z2D|~3)-Khai`yw@7EfMZz zJ#|-nU#tRZCtOyhZs?tOv$O|$YRBEv2M|RD64GnLZmlBu3#Cj*!w-V~bVJYCWbBYu z2W!xAYiQqnv3Huc)T!bQ!>Se|xvoMEO$^R7*31uT=2r6}aFFc-Mj zap>lz@e|}*%aN|pEf3RiM9NIoLE=u=bvS!w+XyOJL01g_|mzq%bz-6e-@DfiJZ4!sqvzu z0h8FPmkDYv*nU}Ir&DL3i<3SB~pi~Ra)fScm( zJ8^oOXa#*0Yyr=Z?bcqzy%%F0fr^igY5fC0?$@YjxQp{tlntFLg4_3sfAVDBVjjsJ zs9w2vR}Ty&Evp=bUsYEJS(Y|JitnP-=XM@k9CH#3xfgEF`}Qlfp(d*;A1zS|hu#(CErPass%PB0!UOy5 zX55zpj3?9HcY?|fPEr;;#pcZ;N$y1~x`$L>g01|+_g0Ci=Ddf<=p@-W?fZVi1f=|C;0j&DfqOcoZEuGN2LuRdS$JM*B!8%Rqn}drFlZfV_;H4c zHWG?sy|pJFu0ntJSs6_qz+~`^gIqJKn2Em9ZVD-cowov>C)jg8;ck-A2#Eyxa+cT% z1;;J%6XwWKIU=q|UUnOJ>jTI?>ToIP?1q?Y%i|1Fq6l(S6%;E! zr5(P198?tz%#9nFtHKJNp(Fh6%E5wjWu-t1`HCke?*vk7qO`Dr5R;meWlV$o-qmSz zjK!^#$t!Yv%IMnqNrBIWla>rNi~2@FwvL6~Dt;m~u(d~j^}!_Y)cmRZi4~DSZkUJp z!afQSJYVs$&#^$Tp&R{GCD|V~&8!{E?FnNJ_-0 zt&S^q9`*~TmfH2^XYK-ae03ZDZ#xrsq71g_#&y~&c3-o2$HQfwm<*x&jeLN`sv<}p zI3w&wY6E@QM<3AX6@grcwCiP60RQcviY^g9j>~lJ;YVYB)4D9RX+^y$C>w>IoNQJW zIHp}-D%~Oz${*L*0(6kbl|HAWmtq3onDe(WD|XTux5hH(y^E{GA_~5esvS<5q5@NU z90uCsh8kg+Nt`~v$4_{}P~O@fwAF0T{5pK)DPz`Pi?i(S*{xkxiN2Vblpn!D%IY?i z)Z?9RtsAR5aOf?T9X}&ksU#jO-&0$r-E;{f=jWb*AlQY};st*7`13U^=Nc{$B3db_ zNalX4L z=xdCxE1%>_z$$VQsMIuiax~?396@35r13NrQYerW^e)3rHazPaYFFyfugwngA~9REr-qJs zC3aK6OpmY9<|W1l(y9q`O)Au!=#TYk81Irl5GCIuB=|59y#_OuT#wHg@rOwk!VgQA zk_>lCwH99+_0-uQ~A9~0sCVLFUoSflzi(jOG zMYu$OFD8h=X(Jwsi-7~L%aDTf)SK85QxdSwUbhFLAZfK!A@RzjN;7$oOw!)oOTO?2 zlb2aLenL{6rS+tA3oZV>Y8XMWj}R0#Gw5<7l&qY?kl^q>q_MS^Ad&MbdZwN>T%Q=hGkn$yMZSohB_V!PT?f9!2bxIF8?s5@*UUBk zeGuN-LNfrLBvd-1Jreqb&Pr`GU8@*zkZK)yn*&TI6#djBb`e-?IlYCyiPN~HoUcK@ z`v;Xa6Oi*|oC-JlYx_4nxGGs3lSKaTR~^m{R|g#qSN|}aSXqho+B*A}0^Qw8uG^fp z&6McV*zAWzoRJO+*Zz#1gURt!Cxg{G6GRG1m^S)3+X#GoFnW@80U zDc*Q@!m5ZpqX5^J>SQ7L(nSqEreV@dC@tm12n|5bCX*O@&C*H8DCB@QW@q21a4ViN`XTb(1oixrNs8)lTve-N#$IcfUA8H+|5Cb?aR2E>)BpgNa`LQ`)&z zk$xTa+oe7Hf%EI#i)w?HKSLerl>J`U-^jnbsa=uzVv4T}Of6Q$rFU|0S-@yXAQ$oa zh>}m6bMGz5d5YZc5()R&U87SaCNUK$=^TDzE+sYLO@WR?5Gm&;xh3X-Q(wJ;5eP(0 z_iLx;U1@uPN+i|H*E+lk=KGe`zfVb2K0Iyu0v-ta{J{C+QMo_@p0ui$RfRXWWjDxR=#1~zP#3q-A~(0e zpX%S$8+$XdD0_`}j=+HV&dMK^=~dgYObP6S76R_52Ju%y`X6&d#LMS?A27Y5{Eez~ zn#^y1I04O$v{F@N;Nqbml0MsqUy@8_6@BHJYb1csl^EXIM@nA3OAp75-K|Dq>_PAt z6CF%((jAHrLCDH4n=0`cEYS%uaf$aaxBnKSd_g2Mc`B*J+DwL)U_DfNDz0O9rWacW572u;5e3N(A!-mlol{FZ}KydSkBoMqEVbvohO^i@4s!M z1y6%10%7X?`dhxI|TvA6ieE>gxg5$uw{@Vmh~iO|N5NZw9Tv{OwhY z=C%b9*c5lRUISWh(InZLPd<(j1>Vb79Op^PDXx|Y>fwwNYBV^jxaAyIOBr=D|2W~- zA6J;BKMk8n4^C6k^zY8N823uHS6Fozg{i#ef8a70R8VsgxCDOSH;e;i8=G{y3VvD* zXi%)C%qjKYq|AB1tk7Ogt}{U(1gxKCsb$Z)`}VSKiE*l{>dI3Jr%IHG$GlL23AUZs zrpO$kqZJ0&;Xzn>eEUey+E(?0zTY6%awBPOTJ%z+>7)!s(;XsXvoCw}JzPFAT}c71 zyW;-wyNn z1Q_L9opPXk(Pae3?lsJ>i-!?z9Z>SCaTwliTwYdq{dO}9Js4%MVV7VRE58$+o3RAWwJ#}!H-!7WS(lpcWI11jR4aP%td%JfMEIK$(@JVv_VTnRF72_Fw|SbR^~SW~DSTG-sXy})O62fC8B~&6Yv_OXyIzlph{DHVru&dZ zn2CdpX6|qq%H2Fte^iIbM*Jo;+T_R3RIP<#be&QD>*Lt~_@7C|sZDW1FTcnx#R992 zw@Pt~wO~GOC6Siwp!Xt)@^T0{5>Itop;AGsb*nCz;Hry^oQyVg5D!95f#z$R-cG*I z3uIGk<$@?Ja=5&YMyq{Fa+~HpGR3m?AJ*b$2AGW1iCBdkFR%_&diBVuIevTUhy{h0@tR606fL2X2a)l zFnu%*m4T(euo(Stib%;;cm3{7FJ(4g(g}zDAEMt^=R6x9tR6V;@QvN>GlaOEoY<2z zRDYcL1+k*+wcX33@|nKH(Es|;?x3x!v&`%<J9b6>b1KRVh)kw^TcOch-5Rj*%Z&b*A~iuGA{5vf zEit|^Fq{gQmT47Scv)K(mc48;K!K*>2{Bucj9~vU^#Y9gop~|f`vzH37ees z;9&{_vn5dT5eCf8y^55V-**fs*bwoEtx|(=x#K2!9 z&3#Q_>K)jnep1mCNln)HM@_UgRDxC?3kUs?UeqV-rjTp9ga{uCPQkmy)a`BPnJ+M2 zi6Q7UUREpo^Ti{!n**WX;}F_ytf2O++ywc1j~slkFg_tS7$?tH%us8kvEftIgo#*_ z>CyBD{*=;LUc;1YY$_(HWl?3>R0^?nNB#8=b#cLebht6TjlNS~tCU0{%c%&>$a6N` ziyg3lby+&ii)88z5w4ofn;yIvJck+Sr#F#1#c_yl#>9Od%Sv=B)+lvIry#uDtEOyu z#j$QtH#Tgg0m~B+wVuipelX!$QGk6)Tt^x|2sxJBw$WkFO%1YwYd`4K*RZ|y+BOxE zO|m6#9GS&r+A(f3^&g+0prZB34Kb`TNOto%75&(Ug>~_$#~7>M-2)epG99DCrAYc2 zpg{%&Jurw_%3bNxj~;$$aLq`dN^3%W&}0o)O^hd9pBY(8w0j&jHiHQ$j#TY`w$ieW z&w|)KRLPEctVvp;u`51+wfoGY;R;L2jgk#XHC60Co-C}IE)3r%FTdMaH5p@S(k#~i znT&m^d+)V_{^uf}vdy9*hQTZkB4!lNBE01AyQj~qD6b)*3iR}t9No&7Hoq`e#}KbQ z!@Q%NXkub`8ll-efLG75?`@5PGH6ms8# zg?9T&xmuJ9)~AiT+GxfDwabl|^_>sP&4@JYR;GqXAV;0XjZAZJ*`=vnT!-IK9i80Z zE~~}M7USd6u!U;GfTflQ)>23D;*)PiKWcXx*l7nWq3Dy!HX}bxyN>4bQqx=d+bmaZ z8qZYx)P;`DJBlRvyo(Yric%z!*V&=Q zaD}E19g9%!6usgLw7aie#@@3@>?I2{$<3W|Q#8nbMT7`^NCJuON9OPdQ5#71FdWA# zxn;a3uouLY3WsHB2w#QA53ZbreY6wC3wdd)-?Xj9!>U~K9^B19;Qk`YUoic0K5j|z z1hWZL{+hGPSyqiE8)>xxBlaUhumP>o1QpMxG{01U9p-NcX14E!90i zHQ1K$pc{pHH0CwGpJWfD?Tsduxw#cCBNfs$E^T7N!Ay)$gHEeTZ{6Lb+4cvW=)ow! z=O~6Xd@3*Gm2Z`wo>I4LK9Min?}jvKw1UTGlk^tSys=9(riPJ?Ne)#W)V1#Wvf1tj zDG2NKw#0p^o%ZHb^1?N^#&a%4Lh0+9f?_X2K|fB8IiElY>#HPW#G@x%?3qt!gHISB ZgcBus$zz-|_rJe3D9ETvS4*0P{2$eNyT1Sc literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/box/index.html b/content/ko/case-studies/box/index.html new file mode 100644 index 0000000000..bead8eb01a --- /dev/null +++ b/content/ko/case-studies/box/index.html @@ -0,0 +1,114 @@ +--- +title: Box Case Study +case_study_styles: true +cid: caseStudies +css: /css/style_box.css +video: https://www.youtube.com/embed/of45hYbkIZs?autoplay=1 +quote: > + Kubernetes has the opportunity to be the new cloud platform. The amount of innovation that's going to come from being able to standardize on Kubernetes as a platform is incredibly exciting - more exciting than anything I've seen in the last 10 years of working on the cloud. + +--- + +
+

CASE STUDY:
+
An Early Adopter Envisions + a New Cloud Platform
+

+
+ + +
+ Company  Box     Location  Redwood City, California     Industry  Technology +
+ +
+ +
+ +
+
+ +

Challenge

+ Founded in 2005, the enterprise content management company allows its more than 50 million users to manage content in the cloud. Box was built primarily with bare metal inside the company’s own data centers, with a monolithic PHP code base. As the company was expanding globally, it needed to focus on "how we run our workload across many different cloud infrastructures from bare metal to public cloud," says Sam Ghods, Cofounder and Services Architect of Box. "It’s been a huge challenge because of different clouds, especially bare metal, have very different interfaces." +
+
+ +
+

Solution

+ Over the past couple of years, Box has been decomposing its infrastructure into microservices, and became an early adopter of, as well as contributor to, Kubernetes container orchestration. Kubernetes, Ghods says, has allowed Box’s developers to "target a universal set of concepts that are portable across all clouds."

+ +

Impact

+ "Before Kubernetes," Ghods says, "our infrastructure was so antiquated it was taking us more than six months to deploy a new microservice. Today, a new microservice takes less than five days to deploy. And we’re working on getting it to an hour." +
+
+ +
+ +
+
+ "We looked at a lot of different options, but Kubernetes really stood out....the fact that on day one it was designed to run on bare metal just as well as Google Cloud meant that we could actually migrate to it inside of our data centers, and then use those same tools and concepts to run across public cloud providers as well."

- SAM GHOUDS, CO-FOUNDER AND SERVICES ARCHITECT OF BOX +
+
+ +
+ +
+

In the summer of 2014, Box was feeling the pain of a decade’s worth of hardware and software infrastructure that wasn’t keeping up with the company’s needs.

+ + A platform that allows its more than 50 million users (including governments and big businesses like General Electric) to manage and share content in the cloud, Box was originally a PHP monolith of millions of lines of code built exclusively with bare metal inside of its own data centers. It had already begun to slowly chip away at the monolith, decomposing it into microservices. And "as we’ve been expanding into regions around the globe, and as the public cloud wars have been heating up, we’ve been focusing a lot more on figuring out how we run our workload across many different environments and many different cloud infrastructure providers," says Box Cofounder and Services Architect Sam Ghods. "It’s been a huge challenge thus far because of all these different providers, especially bare metal, have very different interfaces and ways in which you work with them."

+ Box’s cloud native journey accelerated that June, when Ghods attended DockerCon. The company had come to the realization that it could no longer run its applications only off bare metal, and was researching containerizing with Docker, virtualizing with OpenStack, and supporting public cloud.

+ At that conference, Google announced the release of its Kubernetes container management system, and Ghods was won over. "We looked at a lot of different options, but Kubernetes really stood out, especially because of the incredibly strong team of Borg veterans and the vision of having a completely infrastructure-agnostic way of being able to run cloud software," he says, referencing Google’s internal container orchestrator Borg. "The fact that on day one it was designed to run on bare metal just as well as Google Cloud meant that we could actually migrate to it inside of our data centers, and then use those same tools and concepts to run across public cloud providers as well."

+ Another plus: Ghods liked that Kubernetes has a universal set of API objects like pod, service, replica set and deployment object, which created a consistent surface to build tooling against. "Even PaaS layers like OpenShift or Deis that build on top of Kubernetes still treat those objects as first-class principles," he says. "We were excited about having these abstractions shared across the entire ecosystem, which would result in a lot more momentum than we saw in other potential solutions."

+ Box deployed Kubernetes in a cluster in a production data center just six months later. Kubernetes was then still pre-beta, on version 0.11. They started small: The very first thing Ghods’s team ran on Kubernetes was a Box API checker that confirms Box is up. "That was just to write and deploy some software to get the whole pipeline functioning," he says. Next came some daemons that process jobs, which was "nice and safe because if they experienced any interruptions, we wouldn’t fail synchronous incoming requests from customers." + +
+
+ +
+
+ "As we’ve been expanding into regions around the globe, and as the public cloud wars have been heating up, we’ve been focusing a lot more on figuring out how we [can have Kubernetes help] run our workload across many different environments and many different cloud infrastructure providers." +
+
+ +
+
+ The first live service, which the team could route to and ask for information, was launched a few months later. At that point, Ghods says, "We were comfortable with the stability of the Kubernetes cluster. We started to port some services over, then we would increase the cluster size and port a few more, and that’s ended up to about 100 servers in each data center that are dedicated purely to Kubernetes. And that’s going to be expanding a lot over the next 12 months, probably too many hundreds if not thousands."

+ While observing teams who began to use Kubernetes for their microservices, "we immediately saw an uptick in the number of microservices being released," Ghods notes. "There was clearly a pent-up demand for a better way of building software through microservices, and the increase in agility helped our developers be more productive and make better architectural choices." +

"There was clearly a pent-up demand for a better way of building software through microservices, and the increase in agility helped our developers be more productive and make better architectural choices."

+ Ghods reflects that as early adopters, Box had a different journey from what companies experience now. "We were definitely lock step with waiting for certain things to stabilize or features to get released," he says. "In the early days we were doing a lot of contributions [to components such as kubectl apply] and waiting for Kubernetes to release each of them, and then we’d upgrade, contribute more, and go back and forth several times. The entire project took about 18 months from our first real deployment on Kubernetes to having general availability. If we did that exact same thing today, it would probably be no more than six."

+ In any case, Box didn’t have to make too many modifications to Kubernetes for it to work for the company. "The vast majority of the work our team has done to implement Kubernetes at Box has been making it work inside of our existing (and often legacy) infrastructure," says Ghods, "such as upgrading our base operating system from RHEL6 to RHEL7 or integrating it into Nagios, our monitoring infrastructure. But overall Kubernetes has been remarkably flexible with fitting into many of our constraints, and we’ve been running it very successfully on our bare metal infrastructure."

+ Perhaps the bigger challenge for Box was a cultural one. "Kubernetes, and cloud native in general, represents a pretty big paradigm shift, and it’s not very incremental," Ghods says. "We’re essentially making this pitch that Kubernetes is going to solve everything because it does things the right way and everything is just suddenly better. But it’s important to keep in mind that it’s not nearly as proven as many other solutions out there. You can’t say how long this or that company took to do it because there just aren’t that many yet. Our team had to really fight for resources because our project was a bit of a moonshot." +
+
+ +
+
+ "The vast majority of the work our team has done to implement Kubernetes at Box has been making it work inside of our existing [and often legacy] infrastructure....overall Kubernetes has been remarkably flexible with fitting into many of our constraints, and we’ve been running it very successfully on our bare metal infrastructure." +
+
+ +
+
+ Having learned from experience, Ghods offers these two pieces of advice for companies going through similar challenges: +

1. Deliver early and often.

Service discovery was a huge problem for Box, and the team had to decide whether to build an interim solution or wait for Kubernetes to natively satisfy Box’s unique requirements. After much debate, "we just started focusing on delivering something that works, and then dealing with potentially migrating to a more native solution later," Ghods says. "The above-all-else target for the team should always be to serve real production use cases on the infrastructure, no matter how trivial. This helps keep the momentum going both for the team itself and for the organizational perception of the project."

+

2. Keep an open mind about what your company has to abstract away from developers and what it doesn’t.

Early on, the team built an abstraction on top of Docker files to help ensure that images had the right security updates. + This turned out to be superfluous work, since container images are considered immutable and you can easily scan them post-build to ensure they do not contain vulnerabilities. Because managing infrastructure through containerization is such a discontinuous leap, it’s better to start by interacting directly with the native tools and learning their unique advantages and caveats. An abstraction should be built only after a practical need for it arises.

+ In the end, the impact has been powerful. "Before Kubernetes," Ghods says, "our infrastructure was so antiquated it was taking us more than six months to deploy a new microservice. Now a new microservice takes less than five days to deploy. And we’re working on getting it to an hour. Granted, much of that six months was due to how broken our systems were, but bare metal is intrinsically a difficult platform to support unless you have a system like Kubernetes to help manage it."

+ By Ghods’s estimate, Box is still several years away from his goal of being a 90-plus percent Kubernetes shop. "We’re very far along on having a mission-critical, stable Kubernetes deployment that provides a lot of value," he says. "Right now about five percent of all of our compute runs on Kubernetes, and I think in the next six months we’ll likely be between 20 to 50 percent. We’re working hard on enabling all stateless service use cases, and shift our focus to stateful services after that." +
+
+ +
+
+ "Ghods predicts that Kubernetes has the opportunity to be the new cloud platform. '...because it’s a never-before-seen level of automation and intelligence surrounding infrastructure that is portable and agnostic to every way you can run your infrastructure.'" +
+
+ +
+
+ In fact, that’s what he envisions across the industry: Ghods predicts that Kubernetes has the opportunity to be the new cloud platform. Kubernetes provides an API consistent across different cloud platforms including bare metal, and "I don’t think people have seen the full potential of what’s possible when you can program against one single interface," he says. "The same way AWS changed infrastructure so that you don’t have to think about servers or cabinets or networking equipment anymore, Kubernetes enables you to focus exclusively on the containers that you’re running, which is pretty exciting. That’s the vision."

+ Ghods points to projects that are already in development or recently released for Kubernetes as a cloud platform: cluster federation, the Dashboard UI, and CoreOS’s etcd operator. "I honestly believe it’s the most exciting thing I’ve seen in cloud infrastructure," he says, "because it’s a never-before-seen level of automation and intelligence surrounding infrastructure that is portable and agnostic to every way you can run your infrastructure."

+ Box, with its early decision to use bare metal, embarked on its Kubernetes journey out of necessity. But Ghods says that even if companies don’t have to be agnostic about cloud providers today, Kubernetes may soon become the industry standard, as more and more tooling and extensions are built around the API.

+ "The same way it doesn’t make sense to deviate from Linux because it’s such a standard," Ghods says, "I think Kubernetes is going down the same path. It is still early days—the documentation still needs work and the user experience for writing and publishing specs to the Kubernetes clusters is still rough. When you’re on the cutting edge you can expect to bleed a little. But the bottom line is, this is where the industry is going. Three to five years from now it’s really going to be shocking if you run your infrastructure any other way." +
+
diff --git a/content/ko/case-studies/box/video.png b/content/ko/case-studies/box/video.png new file mode 100644 index 0000000000000000000000000000000000000000..4c61e7440fc48cdc122d5a014d67f106be58f28c GIT binary patch literal 134595 zcmaI6V|-@K(k~oMY}+=k*tRE}*vS=M@kA5bwv!1awr$%JV`BTAdp~kHG7@TFVBm>=+dHt(fA2$^k!^n;cpyn_kh+5f z$lci498A>A!NiBPly88-RtCmxYa!iIt6ym6wl|jr89a`Cno$` z^|vQTZUq84^0BbEySp>Hb1*wNTe7h6^768O-)Gi=3^Z%po|CO^?M=;PB?QU;a$&Z%Hsj-!1OTK& zr8s#-xy0DmB)NH{I7G#{c)7(zx!F0SI3@psm2fb11)AG~{)099KUl8+iv1@XfR2A9 zOPD)byP2CwIXeJJ|21ts>;J9``+w#4Z>-sW*M;N1Vp;ww!}5=F|9{T@&(Plp`ltE7 zq5HSX+@X5kXr+Z^MPtukuwy z=jiBpUu32HYryfQaqXfk|9IvO^G9`a@1@rc=k$&n#SWVi?7^Va{KkWXCZnXx_t}p7 z3#@2gv~_jWT+Z>98t^&F@^&yYbCvBOn1229OfOvglYxPQJ%!*%c--%!Q0Qsak2~MB z1v)H6088uyAQ0rQcE?e&O<{#49!NX}`7Y+Cc8Abe@TB?CEZyz-hHT|EI)A8{=Fapw z9MN4Z^p&I{c5(Q0FZZx8G>7JgZbN2B(HL&+!(HI26PoTfndZ{ zFP51rI!;yMK={v0gWHqx49t<4d38FG*$3+xNh5K|n|To#J2DA96^?GYm>8o%iq&~e zm2yhmQ|KZOJQGg`RGN2E|6;A)|0tjqOA22xVpaz&VKk?M=|7@eY+5#;Aq@o^&$gMh zWkRZYhX+CS;SDQ>=;al!>?9BU1JCSkHKalYuV*HIG@zK6a1u($XG(pj!@C0uf_!pT zHIaf^w%yYyQzXVe>Y=yPa%a#nJ_$U}9IOCo23F|$^3FHA=s*L9)e871>mKu!inU2> z%pObP-}WvfP_Vg&ho>b+6>$n#OxIwV4c+kvi1fLlSLfH%TC7iy8bczOR#Kf87j^Yr z0dKVXdXmAjqH)sbF0xQK`1aMbM?fc%w{Eui;jXO}i|&&aFd9 zoQ4WZ|ky zdV;UjPmqt({DvL`ZSbO)Vk1(zP5@ zBorhPMYgI@4l}xvuTd`tLM7q)!{Yk9Y~Grx^Y8vD+BOAW9)WP20oUQ>6Z-WXRS3vZ zQ)ApM?Tf&OX+wbKrsh&BDeIa0;M+6)Dx4al7u@(a_D%1vAp_wEZrm!u0iN&Agq|1s zH^ayXS?bJgqf!if&sZloCdCPqiArj4!EGPWM zgp-Fu1YryrUVoRZzd3;lCXY-uv020SWSWS^Q#eebTqUT5OZ+f!pN&P>RK6m1Tv}48 zg+s+z5f~#Fw<{SfV7d`>5BXz0fd?p)k|te~`4@<8m-3*qD_acFt))z(rCL5Na-iCg z^JP;W@uA6@IkBIZDZSqNns4+kZ@qoOQlsI?0|bCK8+aNw&*VSs zm?cj%%NOG7W}?wak`Iln_v9*K+=+&rhDV(4c8d`XkbU)k8?MSo>mjm>U^?m2 zKfuY5w{P-1`%M_RVR3jL&*fsLZAZF;`J>;RH_G5(4N>M`w7MW}ko44$hWRl;4V|1_fdF z>aDXL@pu*H624!t(Jm%wkX$AX4^!@p4KD$tnN+iBL=JRd@&Z3<5tb2QN4rWtm&mBg zpw=|CB9v)l$28M)|0bBPB-=0wV&w!+yI-2u$#kLU1%Dcf6swbv_Tvl=8QG*Zoi;Jr z6$vi;?b>05@{CDg*JAsvyBJCM?KY_O4Vn1I^F{JS6Ent9qD5Gn#FN0S%i}$y%Wcs~ zX(D8&mrukOjeu~`aanyo+&&Wif=rpV{ym^u7 zq(;$fex<9>7-f**1tyw#M28LfL;g9X?TjF+dH=cPJufe)CNW6rHaa5=X(UekNK87e zb8oI}ag=AVfP%s(aBXuP4SuBR$prT#OTs?IPxm}X? zZg{mjdX1{TlmFD{Eo$QgcIW_}jDx=77OH0Vid>RP(tcrYt1YsIf3D*?9m8B1)Z_kQ z@v-P)_6-4(xp~M;Qs-zXrs~VCzmYHoKZMP6l{-tb160yzHE5&EZC_}N^ zNuobqb|uB{6*B;8tSBN))?!YI9-KFM4g1o?E?|l%k*#N}Xtqe02Hixo^3Sz=#AMM( zl7x(YJ=xwRgWk?^>AB01CZwI(2F>1dIk(oVC25!|QZ%jfa+#8m$nYTl`d8XxcDZ8F zaFwT5MKDZwr~KKMPvR(p|vdKlgQTfL_n0I$zvZ_EJSS5X=Qpw{i{+X!Vh65dD+p>q<Fw>YMP3_(Rp;&{6`zM1u;E*Jd#Ru;pZcis<5l%fZbx?~_E2X_t_wipysPq9yB`CW*i|gT!oc zk=&zW8bn3D$stzgng%5^Td3+}gRi>E-dD%|r97AlgVpn{74x;BYN2dNM=9W<)7;W! zKD-V$o%**XDyrBJu$`^Njf%6X#iFyQRRQ{d*vgX9+{C@ZTVuP9^4lrx!gn`6K2EmVj}4n&NH^y*NqE{MnY4?mkIYPHmt1E81mhbdB>D z&-r$=VIe%BZGbTTbX;^BfK1v%CVL|SH%KEFzI2<1+_{o&dr1vscTe+s)j33o9GZ=+ zsq9i@DwTGmxjXYvPWfSZfx2S=*qo%ldbEPtHM3eu`Ea8aH>Uby=^GtkxvNZYcXSyM z6@8lB$ZndqUR^`9!Ap)@9a%%Y}xfi3l zJKubu?Q{ao-J>KBcqW$fWpt%@JxP|$D~Ic9jng;76?N=6!FHNXlv+^R#vOM_Vo?Ow z%~+kDt7=685f4}>6XC`~&+hsYSJiGwnzU6Ea@j!boyI5zyH+JP@iHu;bL5#ye9Q=S z-=_f+c;GZJFVfx^G#HG*!`qlDKpXs>IaUE+3KrcHrtT4PnoSlu+WZqS2Q2o|JG_4v z-`P0R(>PW<3^zrcgkP>1EMES;s$lwb&(DR`8bEIa$V^BXJ#aQDdU^~4E7U%f1T2$J z*NLjiK?(+H?!$NPTMx@Z&5PzM!3Q7}OHfZ4$pa}^d7pVsp;`18JSixC^1sJIfYnYK zq~SLkQ{cs}ZR|6YoP2>-Q+yNs{5=}Byq|J6CGg-fP4`zoRz8|N;tY)&B@mG zAD6F=))mD*j3&uZL}~jV-CnX1P+oO0kwsh6;Cb@0)!MCKcRAhIlF=XbplMDS_ z_$tG7&oJ@%BA>??W6pg;01GMT>2-+cTcQ3-zNAJbB9i1%@Sf~%-qi9(*&;NliGD|Q z0cBFGu-D$scMgxf&{V_tci&m)e3%@N<-kUGg;R)*m(*`CytvSHectw95>MwChlYmY z#*dudR-CRsU|>R7IXOf2XsA`HIE$1EHmi+oZeW*=5I}_oGsyfQnrf0Ysv%(EmPqjx z?qx;V(*|{g(-FPikSwhg>?3omrcYPyM3N8Tas$EW$!j{*Co42YSIdk}-0^6vR)eF+i%U5&% zPIFa;8mNFCvlW}hkR6(3ih{IGccypow=I%G<5ONb|GOj~FPVPB4o1a8k&w?RG|8|M z3E4xNw3m2^QnQEtGIJs@i=tJA0sr0f()dSF?s4UHugn-=_*T)iM59RKIWE)}@n%O= z4u0eY8p6isz?P@JOjY{*t5c50G1uQ3C9KGLO!)v#SHTlo=UE)JMh5Y@nzQ&q90=SP z1NPNQ|Cn(531vfHeAY39W2oJl183^N;mgqjmt8{E<15}2auc;a#L%+2~9V|y}Ea&Fh={ertRC9nz^E?Uh7zkO5h1}xZvy%``{Ha)z|g6 zu5f^zuzV3l;P2JUW!9LDjOI}iAZ)dPR#2Y$EUb@Z&x>L0WOU{}(sE$ZISHJ=ugF9C z1U3LyVVen>d8b2R8Y5l3d2@b~L5u8s;~m&6F6%&UM#!83ff}dbyi~{t?zMe1cXM}8f*TP*oZAjLG7JVr9bvD< zpEI5C6{swBD7gTqxhJ1FJ&DT$%&hYKzXi_}fsa;&c?@@rg0%T4T7g`t+LUg@&#Fah ztQkLYi;3)`OBRlO9WiNLpUY@roZ_al_d?#a1JMhVL?@Z8+^=IVFl`gPD^I#v0#1nP z>gqhVf4;5M7&8S7(f#qhr|LLJu$^7k8rpdKK^F#%kS2sxxTV{ZqG(C3g)1SBDP}V2 zP#Ssqt=7Nkgw))Yymp@U0o`r(233US`7ow=A}70UJJ~T4RC=PiPQY1wZ(>W)^#Bs3m@0CySzWwf0 z@BgZI^5B}T2+bG+pY%)LT2drTT@sBfY%iC9IKzv(YQ=`?QI za^~XekVYhogSNFi%%5NLYAZA%=#}RDkpmhe+|z@$CYDQ8fq7r6GQHB5C2QqjA*OiQ z8)>7(x`Mg_(D;^!DRc&G1;o#&!e*RB}0J~zqQJ#Y%5q1UeNgfE_YVBmf^)pf)P`b@dSL}vBrYGwcpTjhv zI4w{7{dUPk0f~y;8gbgYQ2e}Y?~fu*Zi=vh^)yoZ(1x}`@crKu9@R#S2=uEnezc?M z+Q?c{^DPcLRc{!~&}%3~d#nsys<;o@qVHs`RnquzkJ7jRd?#IM+PS=%=NJmS?>^oJ z-WV*aV|r|@Pm_hFAR1LEhBQTEo)wV^|-2PI_X2%qy8C9pK<4F zEKv(%8yy|59zJ8=JZr^ya*20sB2Kgpid`PqJpoa(=Zl{OVGWa#Z(}s0O5m+` z@10VI2x_L@X$J)a5N8RrlFr1BA5cy1&1Ik0MB7<4O#DEUbm{_^bNI5Aj_cG`0^|#= zudiQe>GEH;uO! zjI6THV-_T^eY~{5urHt`keYInErZgko26+O$60EAVLn+~FBUj3*p*w5{Ws&%U7&A9 zvc)=IwSIifyWcadox_~_Qp%aUSfp3E%<4QO6;-7Z*oxj6t6rY3BOX$2RepngCzwjG z(}pX2f)a3ol=o@kgy}OLqOMDI-!NcyI&T|FFMI3u(|;%^bI?pm8^rmFXgXbqlg$1= z{cVnH4pfbT&J|5Uk0J)%xSRN@ey%HrH{V#=;4q+c4b&UxJ)|_g)GFON^F3 znYnu9kSOLL9CqkIC9DD_zTQ!q*7uO6`9os1?KP71_R?(;QT=tss0Hc%87#d)-f-ZW z{>q))s#S!a#mpj0?RPpF#ijf|nMYILpr63r^LU5mb3BNIQ!^FNsT=GLR*mV=vv#v_ zW@Yv0=-vuw2-ystYTx@+1xOT>J3W@!?*=KShRr*X%}0NqJgGM&Rx4nY6mpO9aoW+|n5 z#y{fd%K9aJ`~qDg0Nac7iCCjI?>M^#J0pKVLaqZ%IJXoBjk;M|uhq?Wr%%3{rQnXNyaWXRaxo1FltwcJ1vBA_-v3+OE+thJsk8HL|CkCAXRIjVSCPV}2) z%CSA)K+J1FZ@RTOqM1$oj|s~0Ro}t_ej(BgHY?1sHZ9E4T7zFmsCI!1$A|19hyN(Y z8+H{p2$*Nlnj5D=X3teOj-GpzivL1VWGBQ+HcX>re0!*Pd`>cK3-ge#7O7fgZMWtO zK<@g$k0=&}>J;j;h;|d2Zda5V#f-s8Evfx{q(VvG2nm8f{nl@m^1W zeu7mA@Q|m|AhH|@LLHw*Hce5KMz_pt9byo-A((4KPz8IiMZW&38489$U0jY@T}R z(7h?_6#j}kThg5J8Q(zl|yH4plVwYo=siZdMVU zN~aj4W$F~|EthHHP|}~D5n+DuUr`$}#WE?sFs=E7=(rz3+1qmXpj(r_-Iut}al5Vh zvstXppMA_CYgDGuUsZ}uLmU1TDG0w@4bOaRqNynwS0jO6w$98h?!-l>x?9#kX{ZP$ zneD~}Dn^`O2_sG)*+AmnX3>?9%05|sytU4($IwnLDGspA zLQpfMZ^U`_WcXA4G03uuWo9UQ9?-VqWf8OFkp952?5H@@^#>s~bSG2hJL)u;2I`lz zOJ0J>`_#n8?7*MggRgMnOpHBlwEVO|#Ms}?MosC*SG3`w=|I2V$yRZA*4g@E9SdS10<#|-%JJ^dRA0+@S_xh#fqFe;x`+2G z4Rtm%C*x#)S_#i@TCL_>-yqzEXLT$$dYvz3hHDbs$aQ4_C7~1Oq?*NoumHIG88zQ@L^0kimNJ- zD>CDtBkmw(V6fjdd005<3M4r9fUBSn_hnjK@X)M1M{mB*E~2*;kUfV#b|a2pGZ?}+ zF*zuLaV+L&I|o}Vrrt!=_ zs46dm^BF_pjbPt-FTfiWN-P+{-~A51<=x-v0sN#B=)OdSKMhdq-keid9xUTSlW$1i zNr=a_^PKY4j%y=HX}^8Ls1fw-$Fy-Hv1#5)pM%dEww*rzTw@b`q)W?3Ms&-c2!!im z7GeKOp}UqS=9LMXO><>B@Q5J}G&p})COcYhMldt4FtNySf)dwz5R6!E&P z>9pDpS3uB0#bwzw2pBKzb>*_DrDHiM@>zQTcutd5ng$(me`Mm&~kn>xP^g<~9rP@LnA z8nujo=eoi_y@fD0p1Niv=z+k!j=w)*sK`vJW!9piwa?em1Zwj-Lisqv2&v?I?>H8d z{GO3|7W@e=Un{HYq=EHnp?EdCnrv{q9e&;r(AAGJ4`2vmw}WH{9NT7gQDx%U>+-ld!b@9c_OCBfOw`RwOiSL7Sdnn2obVi>7(Xr0C1vE#JaZ9peWUWU zRZ9OFD5UYS71FxZT$qkDK&JUX%rSuE62t3_cJ-|5g>*0|dDo!#^|`O4b#uj4Hx_(+ zl4E_?g;8crt(+z^$j=@C>+L#tgB$Efg=uA}-5)M^u$7}d$GEtL{@%GggWqjvqf-pW zH-3sC8$ouVLfWWAPCQKt^$mo8TN@o>GX4Z_B6~vmY%qQJ6y1drnpQ1>kP19V!ZCE3 z!h>l#QF|o-CRSK2ROJx8d@UCmsC z+_KvtDFXYoe=eLj469UVvT|w{5mYhxpQ{FG)0qMBYorjW2htJ?hp2kdc*p^no1=4el~81+WL8%y*|NWu=BO*S&z= zMs=$~0vb(QuGYr$#(AB^=!fEO1@|Z|kmgNQ$PpH8xZ)=*6+h9p6A8blf|km-s7{ew*g^`m(1_mn z6&AJwAFeDIX93t0rW=$TR>|0DNO7HR4+pfbNO!{5?%FoYvn0SZ z%ptnum7uKh#xyD4%TA9BNwsND1YU3`Za_aVz{GCrm9mMXJ@x96(xmMu^KLK^Lmiid zNK`mL%5FaVe58M1R_E^F!6a{5`s<>T@fm9f$uoUUMi&EHzvI*)>?v)z?5=AeU=*YBm)mpY?XN?_awhN zcV*W|S5UQwZx-S56`@v5eX87>wPi@Nnpyu_kbWr?_(j2exe(S^r}o50XF#vV%W5D# z7TS=+^n1jScI2Jfcs&iaa3l@$!8JiztJHMsrvHjhF|C7jePbIHAbE^-eUj^a7>xIJ zzkMm{7i<1JfTZuT2qpBuaXFAtYw&05{5MCgPgDn41g9LlPTOvvxHxTc{r>1oFe1Tz zw8OCLvOWlZR#c8qCpyz%EoPY}N$;exM3g2jAWUaO=zoH51oj|&)mmIwK4zd@gT&3bUmqgnvQeTC#};9A|zkgaR9ET4NSSb0x~ zgrCM+aHi~XbM>RvCKkENov}F#$nNn*0%YYi^9MR1#0f7MPV5xixn;JL!Pq|{>zJ!g z6L<nrdg)*Wy@g z5y7@vj!Fl+OJq95Tu^agk@ds^{F98htQKroxP?X*quYgX9JryARgS(&tT;c8g!~G9 zUt}E;O;xpdCSm+lVkV4u&d5mTYHd2#FCAn`b_{&u02w_N)nV$kcwV~k7xaYm>!qu! zu0}nqo3_)rGgK+n<=0z0Dz3=CKj|s5`_(@A1l}g#-d0O5`zAIZwytA1uJTFzgl(b) z9*C-qO_qjp-p#HI()z`fnRY9-g@SMld1ogXYJE5eshpG8d41r0-p%joPwU6#`-n?6 z$0M~yCEv=^+i*`@O(g|hR^a~lD*onwoY*P1G#KQs-i*?Z8~kGzh*&9AKRZCatU(Js zg_`Dbf5!4V#_xlMr{NRpZ9G<=iD=ijoj)6AePp1%6E);um+wX0+kQQ=;WW&U{#;uc z^&nWkDH_Sv`v3zbhHz+LRQCCi0Y+&j-SW}oGeA5oO(0;)8BSh45*L>=?(KN6WoN_b z@Arq4q%4J=1u3F*y85m|v?)zKW}|hTWs|*hoHVxSc%_r@)1Zr2pELh&*=G+fV@P)M zL;=bvZ2_J(fVB+?*;(nB0>2Bc->y@3oX`-|42~uY4U#Xq#Swif!ptof@R-oAMJvSP z!HvJF6TdnI0Hq#(msB8GAH+YKL-Bco@jZ6(Ur4cEQ9q-^Dd1E^C_ zVwKZ6IZk*<*{z*N-7|#qfm2)RZ36jCf13oNLL6*oRa$w>{88P_W)-Of9t^9{Jrd$H ziP)$k6y@N-X>|NXB3Sx2yS#2?00g0~J3+_&a>ji>yTTe~w(0cTVSK#q8Y{BtS-3+U z=*HhJ_8kU0o_4itK)fI)P%1l|x+mw>p{FmnxnLwlq#{Iw!9|Go-KscU7d#-DJG>%R zAja_%;cgX|Li%xcw?8(|AFg2}JQlXt%x9JtkN85`!EY%FuQV)lK8QFwm933tSW0fl zQU3CqNIbSx_v7z`Lc{*4aGcsg_UP-;*IA4U8p~0{lrV|Ikrj1W9cK>aN4x{(0h4XT zGEY1T7W3(Rj>hk2&Z@b7*f=qT4mX$aG)86xUm0zs<)eph?LJ9P`{*+n48+h(7bMzJ z%o>6V%v1*e-GO%W zS1AF{FxKXXi_w&&$}*?!_x~&;k<+N_u z53?uSQ7-Jobj+sAF55hzqObVApDK5gLr%!OQ^|@;f-YXY#-a9uG502mkb(%A0EJ=G z@7pMQ5s}kFey@RgS1+|xt=8Qy&5sl|OIJ%3&3VKJtll@22EHd?_H9l$#9np@_KiCF z-#^^GUyq~2v+XWu=fVbH(=uK%SU|B3xZFcK*#Tm;9&>uCF1G?qa%>qwEa?Bfl-5p< z?~4U(7jCCXOsBG5@u72(0o`j@0u}dXZ%j>0(5AaVaa(Eqy%h6S4eU@^>Q8~fZv-8; zM9XL@QwhV%k@C##PuHdI;KT8tXPE+Vuqw37LMpwke5?#YPRMbkEa!x#DO`WyIX3tcD zjDyu?sa_L^^DHmfr(rV>Bk}vqBc%Ft%rwNK(4?;h?TGdu=gXR73d?a+R7&i7I<*4X zB+kLSV0&jzQorM*YP<#sXZT!g=x3q5WDZm7hqi%8r@a76Hd zOX406BDHS+9`mt(l?^e+##AF$Ld$@iS3b&Aewa(LT9_>?hSP$L-J8X$A1)%z%gRq@Z@L6A2z z8=tx3uz6ptynoZldx0~0=rtmKkv~(#s45fUSDIPAz!9<48$C2%i%6u}de5hdgmaKJ zgw9<%=yvTjir9DY)U9_!-^iLMPlUjoQwW{DVC9<9aM2ZSYMGT}`~n-gb(`?TPR$I6 zf#uk(FjBw2p0H-*7+w`6>&%e>cq=m4vv7hP7t74?Ph$vnN`l2}tM-ycz^)a;dn8Ej z)xX;U_-vb)qh&59T{WG#QJF?wdK;k_cpR{L!C_|6Qay(4vrGQjbRp!Z$KZMGn`{9s z>mnip2O`Y`@Khj9e#DyJ>;@lI_YNKbnhr|zW?{2G|#)(;X7ExoR4&*_WZ01=0Tgw)(daXrltrS5%%KF-~OMFjsS zM#f(>EJ?ae8o90|o0#S<8jZ76M=$0PF(jkPu*Wnr)h!3qn7W`fe~vugyp-IVwtNNz zJ}H0R-#)Q)zq%Y>GMLVnMB@^Ci`;(6?jD$`Z`z9(pDMhsWNn`rN>si~9EtlS`0LA3 zmhHSUbrNT$X8g4SdJj7oN-Uy@y)VT5n10O4U32)h{j(Ro9aPN+*(=lO8;L_xvo zTt)^)YD8V0Qp691ea>TCsz9knmC%^96}tG^SmP{u;R$jvKdHp)3*V!m`%CTZ`xrhg zC`$rPYQpEdM?$A0w$<=m;e?oV=U0gLiUg92#M$4bGE4-aIYMcJ2iT9BuVwp|wUfv%lQsxWC>fj@k3!u`$( zQX<7Vt=LF``wMA+=H`kd^Nrr!!%<}{d|n_0D?_X*E#8t}4C=Jfw+HeCmnMzNoarmS}CgJ@dj!|lM zc8j1%CO&`3JSBHK$L%Y|$=FIB%ou9E^nptrOQ^X%H6wJ-sznQa!~NK;NBO-4 z^SHBOmMkzQByigYb@6%;`E&vKdbJxl&%>KI>`vkZ{)^~47K(5k$>)7+S7dNTY zfycf`9+y{EgRNISp~pcQxJvRQZ-L77q~RtYq{mgf)|F+hXB;*NJXl}qW-{y1q1lFg zT2i3AR&-)&k@g2{^sJMo15UC{c0!*Md!^EUgQ4NJPz{`e~xP+vcF}Q<%QbZ>)edm(9D`|!VwT?5~BDkhf5txhzPAZ(p zxUE$|ia>jVxQ;_ARh65~lf4E7a=M=R_ZjKejrABaH^pNt*2H!~Y45Gzub=71(If5E zinF{U5g4alAWC(y$1gNLowsBmk}%AYmh=1W_12Q@>qV_v-(*jcExI}+U7L$hELuinDb zhW9t2*17!ZPX)q2(CeoAvhkL>otB??+e7ihpI7f@rC)7WmAoA;pw*Z5xKB!W z8fEyoL*!~p1*WYvJ;PzBazgp5Yh;H?$)F8Z<$S$HG!sHs4bGy=h4E~t^K zI=s>*!eW7P{othX2Dk}g^Aaqz<BTDQasLiJdNj4L zkZo(G(i}RGu$cDJ`6J8|T$<~H6f4+9K+!MQvU_t|Yk4VJ4&#(z_nJbS)oDwS5sgn> zTJdy2@hhSBm&}%rmm6{OrWZiyIg_ME-rLgg&(qlxiQnBV57&ARr1C{6YEw|`XWuz! z$5Y_p55v>WD@(uR(PhHIY&2&0K0L9cq9k960I}CT*uw>XFr+lYNWxz2IgpE&^<;jZo@y2;aH+1;QfjJFtDuhTya_hRW8VWpU^zO>@*q3)CNR z+X@iQMAvUgx%x*CYi_CfU=zIIk0*|0yAS|3GB!*yY@v2O3w&^Du0^JJMI&bZOX3Q) zE_CkTF_!i{oRUAH)zAh5xM(FJuWU*a42X{W*;{0wBtEB3=$zqszcnDY>XSgz64I@K zOcqXb0D{^L;~2ik_xY2!zdynT3q#_>x2K1`=}3ZU5lk^e99NX8EcRQe1fjctl$(lU z;i@sS&ETPri@Am%#=?ICaVbLSxcy{$CMFo=)cb+loyp2-Zs)SMr2OERjmn5wk&rlB zA$EpRv~zC;e419XKDb$Y$eQVF3M#-tp_B46R%5{<#uabst0;0;j%MWR6~!{_X$dY) zYxAzIfaQ1*L_u*Y++r5u>~4}y`Pc%mh=#XH^NQBL2QmtLoa+dlt9aV(H`5q@#oNiC z7GTxbg^m{U?b&|0QA4-vnv8e7Y_ZJznIsAzPy{7xAByM!=bbWWVizCZnN?e-D3cnF zw#eziy}~|+Q{XBstPC$NC#c?DB?@0pnI-=&u>oHcT~C&TV!#v zl~`&rKE5E1tXfAZkxbyLy9o8|r{Sp#XrYlQFg-6m530AkG3`AK`nmE8J;V4#wyx?R z-r1n05^&wsv&FXu@hymA<}NJH^H&lDM_S5u8wHP~1T-~v8br5ML_pM$>lt&L)cHpIu znCe=<7ec{_!AIVRxt~vm+$8%jT#No@Vfkcb_&KQkOf-`i`<#(*if{W-~Y^EnaN{XxA1ZbqPNb zPFV|D+O^eW)?oGBtcIP`wBU|g800Y(qmlM@393bf!NX;NIuu>xM_ui<233@l$*lPr z&NxGCd~|yvuF+CTJ!Rs-4c5F`75c54JlSe{VvW+!*+kuPhbCDQTR|zykp_iUL4oPf z+TGVa+9ZBo(%y+PCKAT>ClJYx%*bOIR|C0bMcKF9O=kUQK*sU~rAIpP;HiUg>{xj1 z?@lPYNSA%g_Q*naP?+HOE;gntRN$j%XrAwdUJzRg%}!x4P9|@R3)*^IN}~f-^_wd{ zBB=X6-c~RB4$RK#ehEves;`4qhUvKy3fi%LT)xl5v{mCY?e4GHTIqq?uw;w;UVP_K znv#!+X^$1zXXPz<)`cAWBf#taC7ygNHIf#yZVqhYH!m+5MSq#CS?H|3i(tmPW#JOQ z$GwSUUa2zKGuW9>aNjOglUUO5UPpa6p*J9a!A&oOMe^F1pz)1bp}PA^VlYzKK=y)t zk2UOcNq$32yT?g_I2L_2UX5ANp(X@*xD5l}ZBfAjdWs59jHGbcL8D4u%~vQK;Jgve z=2tyeg1@^v>*Y1&296?`x!9Gq`&z9+DpP_Gy7?KUeYIB~Vq!XB7n{vmIL^-^6e-(e z8)_FdO1|>UA{Bb9$-Kx8Qu+~H_t7E8qC}-tT;DE1?z@-2KY*Z)zJ15ux?akL0&D)Q z?)N6iI5}CM5Gg5vz^LNPIgsmC<;j8PvJMzZkNMg%Jh#_Y3TKi=yg7E9Ct>d5`z{t%Pe|-t(dSq{U=aj+i z_qbuJ`a%>)P^ZuH*LMWrLtd=Bm9D>FyXQ({TGkXOpl^ z%yDi@ch`lY_^4{PH5b67kqc6eb9AkR1RH|rApo_T_%D{#g0wG{m4l2e3a6(h$lx(Bn-?cnl_ z_1ZjW1(a&8iL5nV4MI`Vy{eT02H3-gOxn$8tE&kSo)~Jm+Su=R^v?B)j8M%K7wcgm zz=QB_riEM?yf48@6c1)dc^$!B41UK>bT2M!l+JvWP8*5MCSg%yRpmN=dYTq;6nc_QJe={U9G^ z@;GI{nWdg zH!1MIre{)jaDo>Ttpb>0+9Yx&*WDa8&0MTE;=@QW>+d6Ikk94uC6=~%wI*sL9j$-B zLDEkobGNAg7L=$N=;gT^tajner+OGta~Hn>V|J%KiGKacEIOkyrV6#=EoS_NoU)7S z3P!dF%q`oeCQ4P>TnPw1JvHWTlUge&)n_TZxL`JbUihvlZ^oEQ>hO8-N-{*_ce(5| zj%1mq=EME5-|Yu?@1?0NC{}S71>(m!L?Ou;$add~Me#e9zPkO~?c6UW>EjHL7}i@# zU{3J*GWZJ^@%82z?A$BL?={mec>3dcxYraWMwxK1bsfg(I_7>QmkBlPu5gJ@EF_^I zgcWtsUm*@>CUG^m@pr=ETpmtVP=0BitHKeZnZKeQJ&lA*isf=jVf z^RQ>%3qrr;+7GAkShFh*Tl6!w#0OJD%2?@s&#k*eLD|&X!lp^L7K&whe}-AFp9(cm}T@IcANNTA?RIrFth~ahV&f0Gqa6C1aOgcrk!L&3Z_sl}uBwyYsrKam(k@80W~` zvtH5<6`QoLO-q@3((gpwIwd@*y0z*Is3<$M4C2hV@6-u+Y70P-)DEh_4l+nLU@kKX z$9b{JC{jGHlYM`!xB=MwV>lCEfj_?wZIv_5mvt~XT9tGb_j^PK5S_RO75y$`hsUJx z&^7UM%1LI_*!^WD*1muMW%Dh#l-X#9o;I~rCW2^}ePQuLvMOGPetj6{O@0TjcrlJ{ zI7{jzk|GL>mYz+bn0hW9M7^(!17}=~b7l{d12t<(0Ex1>O4+jvMcot~heaHmrqAr% zjbLU!hAVqeXilLWEuhJ|S#(}+sS2cxb zWQ{c)e;##jl>IVUYB^^gDUxBfTia%xiuCq1wP!G1xtD-s1{qH!`h;GpfjlOA5(d4P zAG;n$`z{bA0%n|Z0LK2YFh_ewcYdNBwsU;bbCEg{=20X(`;fJ8A~Ob6(iL0>h-?jk z4HhfaYE$6Wmbx3-!9}jPr|rV^Y+UHs#y&3@o0~S~);mmocDioe4P-oe&x((E9SWl@ zRGwD+^!!BOl09Y-S>rT^Z_tnVcV|?y!zK-en1zda01=#)4A)&?o_y^vK?|kQZ z?0fwPe)PSUVt8N>pZ&rEGXD0XAHE6Ow{ONDeCn&nQPgwgRp;PabPOj>OyIzQ3Czq@ z#PBj8N`DXE7=DbAkpWCi&Qs*nN}Ay`I?AlD<9-Facs&V$Ar^0}vQ$IPZ{doI&+HOG z&(n&oS3~;lDo#$#V*mbWiXIe2TvtlHWaiWyMR@_X?HI?~uG&OvJA)^mJ|Ju|qtfe% zBjs3tDn;3KYmieCI?iHjY!f0nj#Cq_%i{J$f9LxPfIGwNvP4REh`64W;a92?9r1c< z5^#7uMVx#6)fm{a9@To&dY>*AuRDyxWg@C>;BsuIH$rs__Ba6VjE&HSa1t9WHko?q zdCO$*wUfw5$17brc^z?8E60+Ep7Omx(P@}PBf5;hfioIxh##k~o)?gT4QD7NM9Hn@9 zEDdhLMv9b5^T*Lrp*YO}tO^6)&%&mQP-<5&urP}w*GKsc7j-WJPdULc% z|Fx{2T%?^Jt9}o|OAks@qAvoOfhdS*n|hYnOq@bjLzo{WAQ`w4Q=vKL5`}4b1{B(G zCPpimo!CR)T>?`UvRRgF$O(gknJq#MKkO3zBeOQc^IX zcx+iauyZST)GF5{MD5gDF1y^3Hg=v?O9UicNS-WqC$$c}VrRNgmzBA&MopI& zdY1NG8#a~;mGx{{q^w`cuQBn{W*kR+Um~IzJ zM|OYTT73KN7ce;ZIeg%b50TgF*%7>9LOd{E5z->cV{x&KKmOw{qqkhdt?zs*){gW@ zaX%NEj%{XwDP@{J`+ECu_dUD|0qhyfY6m{E1@=W zp8WHF^B~rb=kfmA-qv-FTdg{N{g>X2?PqPqk>e2_e&lH^F3gZ;-;!9q-l(IT^YP05 zC4B2!kI6L<`FmFY*BwI+m1;WF#_i%1ZHxgM8CVwsp@#7&-C~8Q!`mhIGv%RiDR{UW^ZnV%^#`(j;DKmT{^cNd(X1Ei#$N?zbmH!1|-Z$eg$z z74Pk64{wn|d&_c}ip<=q?Zv>-17tT23l%+PWp#I=w3(B{${3;Q!{X@WI5u<{CL?o= zxOuc>1+Y1i;dA9T8_iGDYIC_pbq=)(J(uVvq$dv!YPS|?Co72P$y|I6v%;_`{QNkq zOaB1NqsQ`WOPIw)W@!oxO`0MZ=&L$KPSZgP8y%>woUTg1#L*|cgn84620Sxc8zyC@ zlBt7!LPc~86*tRVq7g4+=#!E+4=jU+QJlb&e_8+)+AYhD_=ex_1)lkvDBfn|X(k(* z_XMfmS?XNoh});UU%r<`7b9%jJd7K!zXbVQMxvs}AK!<)2acoMJAgf}9>o-S*GrWe zW@am*j>_~`_GaW_TM|I2P{LxRjc1;D3L7`B!x>vP5|~uU+g4brw(-D2FQZbck#kTK zn(N_1M=?6wi(mWYkKndj-htQl9mfkV9>k-Me-GQwJQF|pp*v_n7xB5jz6aA&GtxbN z*SDU;%uHPxBER?2tJ0goP69gvd-mOoK2!%TgZR8u-#@Wa;{fhAnDYW;oXU}W6^wKjZ zQtG!rt#H=%jaW0*FI~BpQh?P4h>DZfvD`W}4mDdXBHfPLQPixk0z(VB|7N-uI!Dq_s?Vkz^?1A z!1%V!!iF35T%mOxu~R58rFtFZyhFC=YgN3$r8?_)*ZR} z)cNUj&Z{m_R*mHu2cu;c#yjcvk)$DCS!ao(6844I!K1?$1{K#SW($23ZH!{9eiA`_ zUfvhj$*VeHb<%xDo_v-(d)=V-WX91d_F#^ClPpEojZJWjZ0;ecdpXPvZpOBF9(v&j z8E)H0r#eY=iJH7NMrd8B`B@yx4Wc+aAStiV)(n_-wfU7tF@F3JWW9N^7X|U|m2&YD zmmz2k(!G56;2}&LJthhaeFYDry#@63^pJ7(v9-S+*X=q7S8m;gokJryYh(=Ddi&}1 z0i3mVEiT@21}@&T6=(DhV|}iOb)_N(h6@x2)2tG}G^B_XhfUiM zNH+qf${1W?CxH@*$1m!Y{$@o^Dku8O$Sa(U2D2tA+_0(HF8`t%^q?A}4X_}3m z@6r|)JftVa^O7`{vQ3u)P`+oM?lq%o%c(BcfF<+E-Hh%aYg1(z+pKAUM&^E-S>MV= zacpelb+V>Bn`FoKaom2(6)5I%(vR`zlZUYP-~{^nhv?kG3u0)M6daJ@*23ZeN2l&)6)+vr-Fj{{t^$iM*a1edprB z0xmdjE&l4$zkz{4@>T|7y#2aMaqm5k6Nub`PyX&t(uDX4$y_kR(W zUVI@w^EcndyKlV`x8Ht)L@#_^^MxEYzVXecY5$MX zeijXtO4YWMB?Zm`k zyzSa^1-Rw5@yvQ?Mloh*=4GN{nN~ZeXt{5sP|V|tfBg{dduSiZbYqIVNWz3#d1g)0 z@aOW!)+1)H%Vv5ox@H4T%^gQ|_5`irJQC5X)D%j*LEDMd^ywClWNL1si5Bhqd2hQ4 z8+M!_^k1Wh?R6PvLnU5mMv(o#O(!R*skz@)w=g8`X_Ia-@S zT$_|^h^Jh)d0<3YSzg}bTHG`_!t-<<*CQ&H&=7U-4zN1O9@|oeI^82ngX^)CY#$aT zBopE(BRnY``{`-*emy8CV@)N;1mzwAD z?D3-~aq9R<0n0vGpE~~8I(eHgI)lOh!@g4%2 z0|s2}06(*gJllWM68l=|yAUVGug6J>C}~p|q_>rAtt2Mwl?aePHP?<>O{YJUMqKvd{E0WQreCJh^1< z3^~h}M%_GPZO# zxb(^o;?V~`gUZx#(G$yIw~`j>mb|U4{+koGH>f5;3q7 zq|q%-sorGloJ;?@W<=L>rLDxj9p+?Jt)vDa4aR)$*xIH}==O$KqSKiJr}(VY>C;B7 z(e{@0vp6ygzlYO>RY`G*=BTWBU%9lG0T(xVs?2ije7?yPJeS>$%!Vd55@1A2hh^3Z zE1Aon2ip|N-&<1bxb8)Cl1YK@MmR7cpu{y4ZbA%`eKejtM;M25!x%X8He5jGf8oe; zB7n*n3aQW7EJ^H}^S7%#jrh6y@fv++Y5g|zAJ~UY`=6luwkp>u)P8CxTBoiwGc}8a z$&<)2HMx*Sk^8YCrpdQiw4RK!k30h(54($u4!-ns@b%{@-RzW0#XDo8REQ){KBTo6 zWzeQKEs@<@(4pvhDJy>I)Bm!5la?Aer1Jr??ekD<9zZGF2Ns?WRn{Dz?vRPHx-I9c z-m?QUV{gR?-^j0!84^ZVU?+!iYO7|Cain<^0q5{cb6C0(Zaojr6{H=@qy-r#MJFt7 zk%!r|O1y$>89`;%zhMuOfkliNJ$crNF>G0>6)=UC-48aLl^R7sb(UBj67b>TtLH@j zgjUPSxE@ji!nVejz(|NYOthTCp_2QItd0?aN}<);6o|Kr`*cl0QZQWRnevt~WG!mY6D99`3q zu^kgiFW2Mhq*YAL-8(H=lOFx5l^s$ftS`?fr!w&aRrK`rvSpCgp%H+oO^oSi`NW`XST^uuxR7P^L9#&gdMYiJvUD`Ra?5zL1CZkx*9o3vkZG}d_TvQZ_wx@$>sL!f%;jJ%W-HC4?p4%s2 z%*AOf#d%vZr5!BLv5mH0rZ4g7Zn+H=iXxUg!@z3C9f|qK7V2BH?cD?@!CCLY8RU?( z4?P3L`f?d%2izG4wJk;Z2oJ1Nj`H<@o)Kt!js}SdXd@!y^q5)uF*6WF*3Y{ zG1`n#a1!udWPz=l$L?94eQYBXCvA}q&pS6nDxd2qt7>cs1hLmOW5m%c1te9JiXo6_ zD#b}Hz{z;)B!o`0H{!HM=h?c;GLNoGcgf5=c@8{KN;H`uHw6Yzj-;C)EM|h^2{H@Z+-k@_>aHxVeH;> z92+(aV(;EZXmKy%SAOtSkk!T_8airUZWby zgBCNntR1-HEmO#~y#=w0=CsgW*jUOqRY#9h4!{iviAu98fZL9+t`w6P#f*s@55vFe+DgXG|Tr}`E^V8)##cUsZ$hoHkKTb zw8=R*`^4cqQ{XgFyMi8CSLNOuD?wq7>ly5P z**O~po0?i^aD=iI{`y<7hRyoiUatCDDRg5mVvpHKWTd?sHiR#ubfSpYGq)ldD%&C6 z-DGXg%Gq;pwkOjAf5*+($xg}Z&zs(~gqv!dYeUxwabDvH;^PNo%40ig$vw8u!?6@u zdsvYZGdnwl$y0N*K3PmFFU}8NQ?8Tt5A>iuH!HOH1$ups$rt2!XE~Kkb|A}R8|ZU> zK9FW!1|5!=m~q<@vWh1%^CfO@j7wzT2@G{z)~94=lF_gR2!nXi^jI+f%Q-U()WmkN z9A72B4f7+nP{g)Y=Erbat8CMcoc-e>{sP-BLgJz795ziqjDf};1Y34oFE_N@)L~>L zRT@?bh4ZOQ7j9WDwdu_;VuxI4Z;9kPM-#mBT!dF5&2FV2c*qY+uQTTwW^AVLDfpyh zj1df2ZaW}spDbd2b~84t-9#W$B%@(nf8Ww^m6@h2v|?BPNTeYAE~D>QUS2XBMkkZ! zRC-RmMj~>DWM+fzo_p0b%*brcTdA)#X=+T8VV%EQIf%gP=oq=jTQ1o~o>~sMOjZV0 z@tpQL#q~S_cX4hJ_uhQ~V?)p4rgy%DB9yCTZ0X(iJ}D`@7F~xLeI{zfT^?_`NZQvk zjz=Coj!zAJ89({YZpIz&KN~NC@RmG?N=!^KZU;hZ+al@}t*zMyBUwRxby>bF; z*9>F-YfJdO|MofDddsEQI5vVzp+p{|GV>c8r8NO_r=4b1o?|4_-8jl?dEC*iB@&f> zpDa5Pu{bfVp^MXRqJxw%^)n6>n3f%DAVoo7AZ0-*@0c2$PmSieBD}!ao~WsE1fHb`dVW zvVupy`z4Cj>*PV2aSY}@HMOI(K&i5+AeO@6`RqHdxeQw_IETFNR?-otQ>-jyN6LP` zN0JIlMelK(QkZ~~b}<94+M!X=HtaSpxu>w=#F@^#b68f{8!^|kS~-F;k)4*l683Ca zopuy+_)aE1a@-7HGj_3nb{B~NNCKcp7OUj4MB0{Rl}uAI zX%}o-63ePmm0hlq$`(!AB59g3%PbNk0g@m{2t-<7vDlnG-twkyUxy^sF`*fI&fmNttY`TcMg8=B6UBsH)6oZFh-oj_CSdT4f^~Rt~6E!O?KqS8Bt2L2!&Ak++i~Ci!72O zR8FbXLVp8c>idJhe$51FTSg}>FO$qrtOSa--s3tmj8Qs~McI*cnRLLK9p5DKFedwV zlSqGCGL#u}<kMx7VG+|M6ST;OmcbV4A^&OBCGo1itdk6-Kqk! ziz}Pi8BLOwtI^r8890QUT^B4n!wI8u0VY&Y0)$b@>~y?$rT5#$v<$R5i}134M2R)R zA^H-FpxhpR2lK%Tv_@leZ>&)!{t3*@@52gnq$r@UXrfmE){a2EN)^fzg|i5c6mT&7 zV+8ech;u~;*VT@BZXwvYsEZT*CsDXc!87;~V7h{qSIhM?C08Z#6Jx)U7u6K>=}+Pi z&jeSWmTWxcS(~yz5nI*SirES(+9cab${EqxDh*J(KB-WY zJVft(%@nZC&wePNYsW57895we1Vn8D#lO3q+mGhHN3c>kg}SyC8^eF|9#25myiK2P z()OF`Y~$wjuVJ?R0$txg<}{ROmJ%FznzIKEwmC>iP(~#rmJxJ`92T(~Ryqq^=MkYD z{L{V}?(uuk#evf!(pET;DxW?D8%Bo>2YD&C%aw{`8JrEr2}aziw`tdpw!Yy?Aj$Ui zRdRzw;z5N8L2|Uh@W?NfSjc+VAQF*C?7-4nmb*!3CTgq&;IjbVSW~IvqLYx6iY!AP z8`+1P^Ob2;&(}KWm$c|31qZ9>_e->3`e-(`snRIogZCcJ{M5-^)R|P`|J)t4aK~Mz zaq{THc>39w@!e;xp;nneiGoa6IDn^KtmB2(Uy?RtsWL~KPvgnwuS#W5o1DcHPcGBX zEz3|p%DIY>%WTS`Erb-3NfjvDYST3`YOPW#%I5N8S9vB8_qVO}Y?OuB`v3kC9)09~+;{)2_@h603>PmiQxL6K#e!}N6Quxu|L1=aFTH#TU;NTn zt;CKy=*0LSu+Keabyd37`gWV{0cGA?$rNJgSu+{bw$(g|&?IJEk;)J zo2Qu(?YKQ#gV>sT81<<;u-3D| zmZ86lJj=Vs%dN?6`dX#k$H}cHFx7sNDwa0=-jcjdBv3YIeVJGs2(&g7Av)uFKFVoH zr8h2NBe|+Z3J6%G*MPsV=8PZx44>CPDWn0&6O9-bg(-GQ(u?T4Br5 zP7fWn@-;odaz{~RY?ujwxyXHttVt2Waf@uGK<~Sld+Wq7?_kc}`G|Qy_uY2_$B!Ps z`8Qt1!w=nw+itlT-+b)5qV2db@Kt}kKZ#(DAn z4iYA@J?UBGlK0EW4}1_eJ#;@+Bz>5^mdZQzJW*pj^Nc#S8nKXb)wLRtFiPZ1gcRwZ z^oYDAOE&Yog3KBZ9!3qMC$^8)>O^WfVQ zNWLoVmA;u3CB+i|{@yd#x*TF6_i5y2C$XjNzBI09FMj3R@}d`8p@Z$a`O|pd*r##u z>K8DmU$N3C&a_tcKmEHyft?xQM+VPQK#4KTeH_8mBs#GTRx*oo0Y4H|V)`ppC>Jn? z?%bzwq(_NY?=_hzSnPPvHLJ zx8t@u-ivnJz_%}5##68KaO~I_sN>c+kdsmclNGy4fkCiymwxqTY)svYxQ7^#^=-NpJ*r4p7-=B)FoJ6jjbuQspA)+NXf_!d_e;ihxT8V71x=UO8XKfBp}DTZa2l-dF|DLJlPk7^R{( z{wdn*><3-Qvq=`Wh;k?tb8_%au)y0~4dt>$bw!k_u^&f~{$xh{HU|VQnVsZ68$G@9 z@-?3`{&F~RY+m%cu3X#1+%#M6=Fo0$;h_(n#J>I0c;n42 zeEts~!&ZG8D@#}Kp4(?}+pRO`^g6iZ=0k#&=b(6KA%_P)a1Z|CE6?HLm1X4fp=+tQ zHj>f~P!Wlc9Gp;a5LxAcm6bY9oY{|m_A5Vzr(U~?U-+eejf4C4;g|pWzk}cUf4_n0 zY94po{84=IPyQU2t}o%yNAJOJ{EJ`0=fCh}eDA53D2Rsm_=oPrO=nNzTi<>iSFf$$ zp$CuSfBGMPhBmFnMt2=2PTYbQUb>2x&tJv#^d#C;4S-Qur3jnr+af4giny1}J?*j_ zM`>Fi_G0mgyP`cHty~Wupk3x$=?YHY{1DDjhPt(M5mB*b4`1%Hv-CdwdF=fkz^#uw zB&`WndbSeiW)#$#NVzQmD%4aH%o7%ajo5 zCG)Z#E-C3!W#&pI)%Q}~P|GAnb&BrIp?C|2hmYfk`M31xF1n^FlmI!9g96H?LmAf# zP;V$Y&_01^W)7>gTzXDWkeekUu^!nu%2FG6ymAWn9{Ma6^qRu9xfUT3+IK~~aQrUh4jsf`+s+zh_n$OJh zj?g8oOE)Ei+R+qE70->d_@+9mSZZuTS0`Anl>)oBMCN;A1A3JL-pTrtIMSrbTQ{jn z$_dQglaC`S2eF$hozAdeF(xUkMp`qz{7t~0VHUd*Ck8;AC8?w!@WakwqQUPaRD#SV zRmyV$LS%1?a)Vo?OvaykOcH#&5=mh$4z8P)d%I)y%`Vu>y=DAHd)Fo@o#D)Da&(9e z=z+X>r77=aAYzkUJI7Hr=xP_RB9;12hi>r2hfWl*L)}0YTP|;nopKaN3iJczJO`3I zwp%d;n}V=xIT-gSt!Q^OW$xQ(G&bnqZ=k-tN|nhW`urx=*0<1Z4bf@S2F?`((^09h zA34v=*^9c zCvho94s|AGv1|P?IHBDg;FhV9td?K;AAScHuGDe&9cS>+?fdZs%J9w{t>CAA^mcsp ztB;AD?cA)!M?ZWLkN%Bcz$ZTQU*my$Zo#ko)4zviw}Bu3ZpMH5cOJ(2))4>lUw#?q&tJk%edI~5)7g_oW1o!c!@T7n={Dbgix-zZG7t958{r8 zA3=*U(|&I#hptZGv!k<-v_esPuC?8eSLy684FYhg)ogw>9K;TF7$ll=W+bsP892R3 z)Pki=meKagO5uzWVAKv(V?JuRO?^l|jbnP5UZX+xs|d{}m$p9Wn(dkBmzARd!uE?; zK1ciDrcYsVY6ktl3PKv^TP|={V~6s~Eea^*6Zp`vPhsKuA7HTl2Ax4whlb;%W+(}+ zZQXQnr2i7a%LYs4!)VYNw#HpHFfcHH^}ISfo1>5YgElH%y586!SA*RF3LfiI9)>vd z%0;}CKZ41VhryW3UZE@ubKT#T%Dq_5iA6)6lEESUn<4l4iAhAQmOPJq&Jvw@*(0liLXwU17xj&`Sn$!Q1UPZ!lV zd)80MQRrXBfwjNHFnk2v>ZAh-+mAlqH@bnx8_(h}RU}0$qsyHKV_W}LJd~cWKZMN) zm%@y}3Exh{4kL_Wl`wo&24UswbTe-$H5D>Q3m!M4QFgd)2jf8^P&eA7QkkF^0=p=2 ztMSd1ZSMwVPwc+Ru$1=-9KHpn&2F+8aJp2f`YMMDZhcHoJF=A@@p?BpFPo9*AEmxB z+CgsldG&cv$H1SSaRQsxkKVAp1Ddp03%1oI0V>4Hw74%`T}G?bMUFPs%9JD<#I`Y5l=pO4u_8%#KuM!txkuklmqDY+t}Lf;oO@q<0Bt=5SK15 z;`x^@;?`SE;<@K9;LbbWCj)bxc!+ywt#+=QmY^1RS3p=V7VaIFig_$VL&xR@ZOFMHT3rj+n3wyA z%lHRB`!o2-&)kRk`AIzS!gW0Td;@cHXYuO!ZQOI$e*E%(^$1>i%L^) zqv#Ln9RWR+r=e>N`nY%dHB5i6j_>IoML-ooZ<5>gbRA8)|BV(UZS?o)67A!V4t=&X zu#t$XM@DrPD3C;=d#EHS;{;%;T!o^2*u1fX=JgxcSYM<{x`mtHcR!Bpqw0y0%5eWQ z&TaH?adkk!DUg|+;(ncp_Y!(=BD8$Zdzc-*M3s)hCO(Kpbp|abvCl$|3)|1*$o4l; z8Y~I&%qmR>i4=dpvrl%rOAkR0w~bTDl8nMBRDeCS6C5fA8H0~eGai*;gnCkj<=_U{ zM1CY{D%Y6K+M_CD=N04Az`aTZ>6W{m09USE782wPWrnZ5wn_oxOXBa&_OZY5TVIufi4j6iKYN|BAzH*& zVi9|k){TQ}D@SxnRC$fnpt3C&s}&VewKYFq#QAe;`1ODDXZZDh@n7Pf{NhjGcmLqa za?fr*brAbzL-Dz`_PYF0M6JBf&WZAj6r+rmGOEdm8vgR}r*QS!b(}rL!i=XWXiQ*v zbAu|uKF-kR_{x6%7r%@XM(obVy-wZA2#Q93sp{&g+?Bn;>eR)G*t{H!VZGQGcXru-5&c<4D$u7cWv^P z^9+4gx$`rP;-Pz;7)0w?|AGnFdxK`sgC3_dwRd#>7O;Og=X zZ1>siv5X2;Hj}vsXHT44J zV$UL4A5zTzCIj4pqWpqA}+s+#j*q@hWPg4z7wE~&}L(4m7_{!S%&O|d~Q@# zVB`=oBiXn7?l{6LuV13di7JU_>iD(a{C)iF&;BI-?KfueM}Pd6_abZJ5 zBDXfz@N+-+0sJTb*+W=cZ{m;t_#2q07(8(AomBPBV2Og{i_a}lwgx=<&^`E{f90p} z_3yqe-s0WXipXkw;zu6D+#ko>*fXXLIxLNta)bc$ugbe~G`J$7p zz^itf(%jtC0^a+npTL1r`$h7CeOe;XFgI5JDargZoFq@WO)7u2t1OG~FfwvL-ptJ& zZZ_yzk<%c#|Js3K0@2~hTf5#2)jw+%7X;Z+vq=Z@&Zavx)q_-wCNpQT?xZ>iCfZ3x zNQDCn$+z<~opvNtoRK)P?1xGeV}JW;lrEP5wx2D8BHqY7{9I;xuvQcg(HS=U-O6zc zk3WpFlx;_w7sNEn1im?8_gbIf>U2^b>z#wS6ksLzIP$Zz&c_3|d+R(-)fpDtLL3zw zr-NnlQ!s(QZ-F19I4(y_!LUO~$jRXaRA2rwu9Z%qLwo82)e>%^-?_Z>2BzX3PSpyO z5Jh-%ahcXi4-=HMwFer5f|!`l`VCO4lrTwuR_R&JlydYQEsswPVobAMSG7hFu#Y}# zZ_($D77pOR{0v^AH++K|c7kgPoI*2O?mADQd^uyc;K2k2dDX!zs|Tz{=$Vh9sF!h| z@dV0)1{7=1sla2Y1dtRkii>_%TTgI;lyzV8vxjsxqFiNRlpn6Cu|s^X6CEQ$Z)sMo z6O!L}#57}g`Lt)OGaJpp#-y#Kb_~KX(js^VwngSG3K$^6AVx}V%*tV?tdGPy`7aZI z5sZ=^cUJ`*X{sedO{;He^Sdx`3}!2j&)LXOyy#UTsyPthHZRBTjK_>8cY1R)Uc+9B zJ$BZxYG*j(9x9Vg-JuscR(ZJ;rh{U)>GVd*M9RaaL@aWOIzXhKmS!ME_d;p z|IgPb!>i)qhd+q5wINE?H}I!lc?t6iU%-F#GoQp=cRq-vb%THMkAEA#^ZCEPKl=HP z$@}@eFFb=cFD{~3DahclP>jT*!NN67LUhe0P$|e4DR+t4<2p!(^z<%UbkD2PWgI$n zn5v=<`aDO9qKxL3-lK_{ikD`WfXQlIDJ0ppNvx%{n?*(x?Raor#2juRL53K&sp2+~w2cPg*UJKx;z=04STZ%KtOJ&j{)Z{oJ4Z@>(f&=J?c&>2qodlXs^kHLVh zETCXx_m0?4%Rn7&PT0rs#;X)K>R6$mv42PjTW<{$`55`BX}q#@9n%!3Za#1jSJpPf zucpzaH6Ha*qtYukq$}+WsEW#CqFhB#sZiYLiS>0-4_bs1BAa zB-*v?wM&*lA!jOdO%aL|ez5<&t(Ujj<$b-Yi`W_HvN~g9PADl1H4I18Y_rvkpRVu- zIPlDCUiv(2rA&mg`8e*2)1_^R4W-}gb+1@t&N%f}J`}7enAd0uGj5tbm;|H9xkQsJ zOuzT8;6SMyLKUPxi_BA^6J(XfXp7l0mUF|ky zZcIj`>~3>w1D)n38rvJfG2%H7E0EM18x(YMn4a2?5^XGYs%y2kv45^iA2sxQ0oK>~ zzOLb}d-hYtR>U(;uhC@>aM#@nxN)^d_o9tE?l_6ZzO#g_&On&q7LGo~{ZJWgIG0ew z!#3($t6(eaAqA4sObOFRr)ksnCAf#KWviVPw9cRiv+!PvgbMZ+i5w9F@#rj1oU^h$ z;wKYXlPjey$(UPDVpzuRo28OiCJ67Chs$czrEv%Q-E}am7-Z-GF&@D+9AU(Z-b3Kj zwp~z>@jCtfn>rmFqyy1%JG7t`LYbVdpsC8%Wt^M11HI}Y z9NT;yQ{C6l3ba$k3X^o0HS{7Z#g1 zL5K4_bW!Z%oTH(EQSJntWcFaU9E_4 zsVF$`o~GZi(b&XFeTy?7M0r*a*=C_7%#>#k){fyyS>uMe7t z=5YGeuOWW(yA(@G^!!RTNCfG~LR^G60X2+KD_GQR5*sAzM4o4Vs9<~ME-W8>q!FIE z{nMYL^-vf^FQxbrli4uY!C=Cud{-x_Nwll7AL+(WV}0VW7Qo1+QX^8@e{KJnu@CHm zrrhg$&1f)DS^1OQN1GXSVwLo~y}GVvGfUVHe4r1dVv$A6Vw$vZMEo(A6M1pp5@G!( zHkIlR+H??i#IJ@w;@FnKeadLKq8ZXgV~bF++!enX?d${W2OlKXdD9(O&RLNUu2|F& zhN4BJsJx$>oyPLow#;0VQwr8Dqm~lffe=Kwlr_MVNNUn4g;ziJoe84*R%woTETlVB%H<&Fuh;9L@vzP>Vsg0Lu38kgU?MIK&;`tdT2|OR|_xqya zEgS}6yvMG6XMr!{*OTo)gbw-T_08mGO*{?>$cRF!QNI?th8fC^Ct`d0hr&h-u~obk z&FWeD_ZGt5x@5>+>n(KE1nay-R4td{>&Vgj=oOBmlg}l#v>vc#+y@^g4SMAu`(v~Xc_6HmN(9^bzBI^I~hPHT>W zc_oi-aUkomo;!-Q!#{?L;VHb z8@CzkU|TsiOQ-+g1SW0y9k3uIAEfz`xF^OUYtuug-xOy*7GvZ?Zz$~u$4#J2OGsJ9 zIGC@qPtKHzX$2A_Oy}ePafhXtN-<%N{cn0L#7scUWw|*)W)2Y~kZ4iA>b0@3Fo_4= ze*&FvuA|#2;^2WQHdY&W|NZx2ad{Cd>ucE9*uY)yeE?tjv#;aq*#kI!@;I*ESQ2{! zo)LB1J=v%|dd~&A2SZ9>UwG+FcR0tGnVNN{OF=fe=HRBa6Hru%EKZhYZZce1JI4zI z9$cBpVeVj=Dw2WRXPE(HxJWwdqdo@NOyp>k-qD5~kqI*?cWT6E#A)-2REQ(@TKz2w zCIjmc9>{?jB!1!PaDF_X>5Mc=%BI)aBwx#dmyQ3Nt6Gw;yY=APBOdtAt*w4^2v@3) z;96-8H_6Cwu!)_m(=pq%SqrhOhc#bebU;MKX?vHIRe zXD$zgb^= z1u>rsK|W0jRH_V}2;Vxs9PbN5)nt9*EY^>F5StbH9X%r&Y8i;3Mmot*nc0c4(%gX> z6{q@vwU#mr+ug{5E^v*1#rB=HV6Q$@(U{L~A=^@9G>(OaC z;2xpGVouiQ^X0oKV^%nJ^-mCY+0(u3Fkguv3uVhxWt$cjy5AMRLoi2&WT`4j#NG-R zZhoSi#C(Si>~9rM1II#~qN;Jwx1le1!KC86wW#DtbaDEevemdcPhPcZ0XW8Zsm z?ZF>IajGWWEPl`Ji7LMR{!ioHXEbsbUl1i>)&(?H`$>F5Vp_+WV!nfg zAFDaseC8M~E^T0aWm77jA-!OS_C}w6Cg(kwSYM9zPm3xWCH9U6uUx&17cablHeLN1 z1)1e$9bvJI1N&KDAdi*pKAyR}gnJJiLv-eYXiS_!r(*;qJWn^Y+lX+Xupbj=K8iB! z54F8W?>@>39%R%FRC12UKdI^&)=r~-^rN^jaZp~fW>I-&&qbZhj0D=Jo$-&th@Zy~ zE~uCt2&$3R)>uHXnNlnd-ZT0C{r%5K&%DbSNM$(=%tUlZqhWjwK4u-O^*)y6CVOEKb9f6I5 z5CuD~Qrev@)VJ%{-rl5aw&&U<5ag5BU(-e>-~^up7!@WJ41a_Un#hNci6fDAVGo9G zw?hX-;Lu!w^D9w15)6yA2(Y|V$CWFqq6EP2cXeq^WzD_X9IqaLb*f7R8P&eM%5YASer|Y0X;@khu!zZJF1o-X|LG9xJeQ*QM%=!aV|`* zB>_rgp5wUjmH)aoj7x<%%nr<^cX9MBuC#Kp_R7kHGB;s}cPl!9tEG=%Eq4rbq~i>| z*t2w9-;LLHzpbrcsK{uri?Okiu-By%*{ zBG{s8;f4Et0uRvlZ(Vo}n$^%Hv9f_XqUvON|0FJ#OZcOU*HOvM;7)FH?hmI9&z_Qcz`9{E(4cL5vMbq@KF?Dz0s=p_Hp)rZ$b!hfm?e@l#aE zM0n}d*YM5Ni?~rgfVsjgn5x}|_Q1Z~NccGllPvADVeoS0B#!O-2oCrEoU*T_SffI>oCDfZasDv2k353q%0bkJR?jT3x^_C_?T}qAdIW8yvcnTOZe3|#lGryhAX!7`HU;BsT#A#bwyptN1$^WHGf zBb!?WxNI#PDhJDSoC{&W=Thw}cc_PvN|1Echg!)%SvmP{9srhVnkN2sREm$ni{N=oNmA1S_7;@HJK{Wt5Bbv;A14 zVZ`@6wp(8mq%()Aa1#V9kixlp!-4B71!4othHRWfv|3gyG!T4#D2|X#`dlEIQxO`J zZLV&%WoG1>XR?OHUfER@vg%#y(aLuFOgzkROH?NCsiaLQD4)szg{dATZCVb53@*#g z8eBWgw?%_75JNg2R0&cqUE(?&7$+H`k`{^^l)09W%7A0i1~w01`FTlmA&fE_dudoO ziRfvVNNw2o6v{kIC5Cw!;;>6xQuL)=qaqJuM(N26__90}T35LRTq=JIS1FK)29%ST zKTAd8Mf&$Q=kCUX4BNf>r&M8XAuc#bs|2j->8N(nY11GRBor-ftPK9_YoW+H+A4YHA0chI8 zd>eJolKKQ>LWk;VrHb!9@Z-3HU*p1g@n&J0B0*R37&{l?)QO{b@tHQhx!8anM3^dd zut}S7xn;1~<5@_D0*#`4gx7i>McS`Ar$j})JLn=`Dr2+JMW@@xso6s~b>Ik&9zKfs zg(H}moyR#?8Z= zLV|HE_P}J+wu35G%?#FP8?iD$!RnbZ&%}%j-gkf$l}fPdNq5U(%7f61aw0O0QRZzS zQRZD8;K)c#q`lazCeYB5pt3R!Ka~IWzV>(WTltXBJLO=rTNA_VrVC~gCNx8Lr5Ov~ zmbAzVrFtW3$j0lIB?_T|tKhQS8V50DCUjN~mr-H-dZr`BWlZE_vR0JKTPb?>xeu)~ zBv%fe&rtBXTX}oER@&UcjyZUCnp;?ypTOb$(-iy)b`Y(#1R5(um_l4_+fK=XAIf~k z4*8{3$6nWTu&^J8aV?yoHqHmd1}9OF4V)Z54{`%XjWar7-uH$&UAvydnu{vxhn>a3^_0*yfXV>v}?q(CRo#D5p9EcJa? zJEJrdvqzh6l;{{H!{0E~LUO~;K3_-1{EE)-9ZK0?Hn47 zIz9hww|TrhuSqrUGER#2cWY#TryS(FMC+tfWa+6APIEJtqL140wBD8=P#~Zb^zxH!VqOkW7rc zgnec{>%Moi63Plcra@Cnt4M`8teaag$lrpV+n;AZs*p1P}s`)L1! zm)XcJFO`KL6RQOrs8+E*A7VeF55p=brl8$iL$|Sw%`Tf1c2J6j$kXO2((5-k5L2Mq z+}c9_Mi-@Wkzy1Di&Y99#UiTwEUDVpV*FFYO5DX}yNzPNmf(G%lH_4OxRT-mpIvn; zdVu|%SFo@16jg?6@?KMPtbnuAMBd@}Vy2xr={6FhRnctp`I$Lbyypl=lSYLt&uDjDTO>MqcVhke~_4@jT-V` zlE<%faty`2Q(KDyZCpi?(P41Cb~7Sd0|^WQ+vO@r+&W#0M_CUS8;kY-Yn(OUdtNapPEmD|v%WkY>12?Z zNANRo3I+adYiGl&GH69)W!tsMGG^`TDSwzdR3)qwO)zGkZ2umQ1TkB|IJUe&a$f~? zDi(>`>DVV(#Xdj5%Vs4Y)ibr61b!dwscSw&oBc_I8~pGp){ zrVPAlkDMA^@+^IAH7MfCvv=cBy2jZne~w;%11fU1;d16!o)nWFi)I5ynvV&eC7!(> z>y?Uay|{J5)e3ig4^bewW#t7-ufHN~B<0iy0yU1;chwocCwhPPHeNyQ^)}8=J&fwi zehbkZDmI$68y$vk_bH$l6!Ns@cwR&6tI=u+oZoD3N*5qrU&bP>(TI|vVYiR=poMLE zqdb(SPz05mMypkq3NUiAMy!V1r2WUL!25cyVXpfGa>L7s(z4On1U>a*OeaxRxuFoQ z;=ty25MSzJaOwjri$T|+9ofk*@E{z?fCb~O?fe8|L-%CE{E(1TO6`r&GJ8}$QkuCw zru5bS9iY5^SOMj&rZpD2rxO&8${>i;+ldsoFJ_fug>}G(s94`l2s$d>u3Lt(>nCT2 z=5nXC8dD3fBvpY_N;TlwA zzOCmfVIO3XIast#vD;pZ)I!)Z!BSH|buN8$tjjEHn z5n1?IZtu|d^yw6Q{NVfWp|Fk0m2X3LXbndBEUZNixglL=#WZoU`6TjmFh7}p0R5tM zYlQ=%PEou+yYV9SZ#+rCXVY3^mRTad`7ttX`lb z;qbi(kKg)<&*^wrkiM!I-4Ed@zd}As0!<1HlY_uzb&kTk>zqOXduVo=L6H+PH0!OIonkz)eDGCy%gRS-^`V=ZU zx{KBN3R(;S&KCtR!J2!*HE}R_?+htiGrC4*!}pma9Oj{^Y9C0aDUUch2sLvb_MJUV z*FnLpWnCl>^jC4H`wf%_Z&MX3T5HTyd`vp z=bqK8eKVYCi#{1;X|aghX_4!P_ekf%me)C>op?jWcHIlIOFP*Uv1y~Mu-`*e$z$Qv zJ`|=4=y8VcWJ4_fLM3eVG`OgA9+?OlnuIWC0VCnABBEvt2s8fvf}~2Z$i4*mrz~>H zzF#{dD&z~&?u*T+o1<}+T^Ix-92VO?jFN0S>sTuLHn1*|#s_>yL$|yXB(LWKov0ho zpBw5V8iP4$xv->9E_M!zC7|o@)_icq?zzW-ZOv@E&-C*S|)GuC15T1O}qeAnP@@zyyl~RL+6Y z&a0S%RHRdOsiO7+9D6FqePL!l2&4lz}X&_8+x3p$V1t4ru!S;8IF z0;b9=64R#u+eN$A5dFNc5~9)HrhrnX>&$@#``^IS;6SqgeXDvi zqO*5NwPZGz#52hRwh|P~fs^zv(0izLSNjnUAX!eM)2q`#(UYu#2c(?E2y`-FY(M)d zr;dz%^NhDq_T+~E891;>290va=)h$zZ^Ja#X3tUg_(sc0#!D;C=%h^h_i`TceSE*k zigCJt?ETE8!;(?v{VQD`AC_DV&_*bhYG}1-k#pZRnFF|kM~3A&8yr{@j2CL`r8;bu z9yrf<7OE4^lcZhw7GMo08zbtrV*-o87$?Gv2r{O@$j98xIT&dzn{W(PaWjdfEcmG0o zadt1D(W7IuGY~Bv?PXke>}%KxK7#U*gMzYT@kux{ z(JW7o@5ykgfW^vw)UU0ey0wJntj7L>hY>fa(x=3|Qc#!-Ssq5?%+VuQ-RfX*yJ3-s zRAo4Nc0y9-GjzbwE3{lQIY1d{C~KHehV*ZGFQxaMz_P2H7E~XnhTpP#8?J_UqB_n}oijyEUYkB!29+;RFKjL+%HhO%~t%DP2>~)!>Ua7E+zCx7M@xNc&~8vR}Ls9k=vI13&Vb2WLRKK z!nY80TF#1+d7IU#k~WL6*?XZ_62`qGDM5(T&DgvxP&~Q*AR{2?P)y?lyYXOR+dV@G zesF{7Kp{B*)p%j1g2Wfx*pVb1aCFSnayWAK2uf2WbmD#@N1rtAwDt3_R>`B+rS&M1 z3kfkSB^cTu<0a)YLtqu`YTcHa(uHF_?6%LwbI{??PKWb_q9wcPah{gWbP05_V|+_3 z2(sdNw(W;R=+%oy`olOCaJ9inr>_-SgdcMin)(bnW=518miVxyz_DPqaWJk^Kq-m> zZ##4xzfmw0@kPQ!PPS#$8g^WNrK(|e=mpg)^cmHja?s+d- zbng~AJv`9*CJy$Wqvyb>Q7$35GXG8Pj2R%O*$d??UY~y$*9r?*Y_dRP0W*gVq0-uf zZmrU@iClNV1{7<-5%;;74H^El4{5u7r_q`F!0wH zOx{Z4ESVQ&<_Uk*uod{gPucyT*UgZ;EYHDazb<$=moV*Q%Ok0DbM}e4Dn%QY$-?Y1 zTiYX;s34gQgs#oC^T$RBfX2=KjMu(E))`OjY?1mkV{8J#3INe$0mpAUj?!!eZN`4O zxuVIm;jAf{8|XXpCtp#sO7jR%!9%+PH8NB2*JY3bThtyZzUli&XDKFuc2kAK$_MNz zP8*;7XR_A3;_&DK{itnqM&rxPo02-%=n+`3B@4$%TQ7Uh;XkL+SxVq0nGJOsEj=Rv zqf3YHanr&5o$sN%@-&?}0rJxa(aTvXRW5SGLk=cZ@rOPX{XD&=$~@*u1r(Y~bd1$S zQb|HYkeYx+K}@l#gGjgS*e47VG@K`A@&U>ujLpgS;M(zz;EmD**29O8Gq+ZEdSo`qITsa6vH-J zg&r2yS8-+eDhB-_N|l26vG8=HM8ULD%!$xxr!&AteS_9qSAs{8_Rv&qk^)Xqcvqo# z6O|+w?#m*# z5^oBbB8Iz~ZoS(;Vv&5QTt<%t8~eRvXzYi{T@4Vp{~#%Zgm9hEAQ z_W_kwEyfSv9TYfpu&Hi`H^rl46U=W+7(lc>y9 z(eAdTKNGk^7@6wIN0uP5w>3^VIUdtQ9g|DhA+R_$FH$@2k4lhdo@0}>!?ycs!25~* zEadazu$lVIAgwC&xCFjRIAFeIXq>I$h~$fJp=EU66jC8f?Kn%xJqawjko`)w^(m~Y z8E}jLHu~}-tyd^p{x(ea3Wi%OlT$%BaX<$0LzBL?Ag^O3NKG!dT1-@_%wnceLZPvU zVA%4*isb%S@}z}aSzcS11wLEVVtTRq#2K8w=|{0xt4f~0Ex7Gc4HI)ynBLk#+}e_= zLr{NkMqlpzR%oqMn;o>)R_WlLLoiQ=eF-AaSh~1@#vgtWy+8W{)YdPcSmidU67#LP74=Bwfq=+^ke1gWdq0UPt7JiunriQAx%Qd@l3( z9GW}@wgZ?t`a#_Ep}&j!KJ$-o_#>Y}Z=Rlg1GsbZMJ#lmpu(&U7H;BM=+Muvc@6P^ zongbpqqscx2rd*)h)fEvnsUyF!3^8*bVgATR=Iw7_{_tf>-8Fi z7DeFL^kwmJ3o8ioC>N@zlqs0d8wg#Ci%poM2K5i^o%qCKr?;RDBttt4<@Nv%NLdF; znCznFBO6hBi4S9;YRVxO#f#gdcD^IXAum|xi+SZS{CX^-RNr?6;yE5xjw5#Hr4-li z^?hM<28e2sYAO5Yu3GhG6gE0~rFslTo0^DONUTog0*SVgO65D%_wkO6N}=H~*dwJY zNkEXS0q0$z?2O82i-eL^k;+Mi><=d^7Uv9=v>xMZ>O>}gbjbux&DduCDsWyMZ-E{6 z?G`h%a`|Q~n5c})P7>sdt#$_V{=!OxV;o5KO`tPqi$6Vd%8m|H8c0&FpD-D^`}S70 zDj%(~?Qaeefy>mQIt5nkbSSkJ11_k#VWaY6Q3oR6c@^KEe4!-4#AFg1YY|{F*W?fU zV01p>0!ku*pZ#!f3S?Vyo*Lr=kK?Gga6r!>hb?uM4y|dNpkVRg!K*mce+)ylEL*ue zY#^e8Dh{i3f*n9FvhA(Nj-{l{#gz#^gH8;oP*n9Grt?MQI~(-tI>HGGR4NJuqcVb= zDeyNG5Jmy@sax^-sgF}2nHE*Z8hx52@)=B3FU+DiKa0r@RUVCHi>h-iG>4!&edgOD%LA%iD)G0`JonJvt+FMnM z&gD{-vWT*3Jz4fbV8@gO#H`uhX;4|$u&TL+H$#sW(7+*ToNNfU z3mjBAHaa=brGPd%IT(VG)|yEMZC=nXDc6JW`OzkwSlp8eKfaa zKHsHHuZj_l-f;@$gR|(yREfnl(}#nQjwmte*!kJJ`|XH<;>E7aPVvTcAmyt2(c~so zCGu;3-0vi<(6YK-{LqK43gi1J7RsFIdeo2OjEOl{=2gQ485iu>r&&Lt6qOcds-QHG z?3fez;3~RW0}R;CHoO)4a+7#)a1N(?-=VDdGI}OQ`#C4Vr$y65(4fORm_}}PUeH|9 z-r~%-&+u4ltE`2Cq0pvp=vMd9fo@RlZc3n3Y7g9|5C^TMnW4OUyLL0K9(fd3rsmP7 z*DPr(pTeUY_L%Qbv9nd4!sPw~sBJA`xV;XQvx$sWR)p0!i^l>bODqR7ICl;8uYLoA z$G#*oC53z_6;_L?+#Y@Kz7a5p*5VVy@FCLTV)D^ffjQ)7IqC-{hNMgtvV~L{=w2^)L(fH7hb%8`ijQFU;}6S-@}BuLDvup zdtJ+V%t;Us%`M$QFE@p?*++4i0&7py&F!uXMVriO{ISR+2X@n8`IVP3*}G=fiwoSe zE&Ab!TR!qRg}6Wg1?_gjY9)mQK|qwt6DU#{#qF-Z%}l(A;Jab3n_ajO_P`lPP1)}^ zvAMoTAIs>o-vxcH*YD7c?}>mPXM~D}o*u?&>;Ot>j||hv5Y&_6%m~C7iT$t|Ypevj z^y^1>4tuqhGJ(W6K|{;QG08C8&wzYOFa0n({_YQ1t1@TOHkz>j_H>-CQy@a}ISbb? zu9B0(L}05y=KU#TJ0_`Y$Gd5RWuzwPIrM3Ba6lEfqWs=i{7|fEk|%-%c5a|;msS!1 z4kSqn$n3#I+U0p9Z7w^A-1`}g2NP@bwbTK~$=SF{Ndz6^*xg`jZ>ur_j0x&YWhGNG2tlf*E> z3ve3yOuYD~$*BIC#G*HCnfd!Evs9!$G+ACtT49aq@u?sfotI>Yl^$YiKN|}xyJ8k_ zyXoTW;5&%oOG19r^m*fa-I<@k@+gDNHS|IXDARKygVO7X)j7RCr!N&+c@#!@wfRiS z&tWodBj3FtNvH2js1d@NKM%B^yw(ogi%SPjp=}jkEF9VCaY_4zVZCh%1knVh4<1CJ zy$RD=l@_aq(TZJmSP-Yf9+d(cubsp2+IdVxLs41ivbArY14&QTQIF2FKCedymG85a zF0Yw<4h^bI7VC>xT)M&LpEx)>M^#Eu0#%VJu5!6dYcq!p3MRci2LdHtJ)HPA+bwBh zm4h~F%@vqSucG_LOBie|$+guc_tW(#%OjL?aEKhOm0mQDjp>K5GI1|E#>=8k6|rq{ zMKM*3#>(9uPS>Br{?>C;(R7@ya$+M+!Q|GDeU8>$fe+tapGq~ls2o)x#p)zwH5153 z6+z*ItZx)XBdVg`PmkYc(ac}5Z3B8k-7Tz8RNdHIqMP18x7$LC0$Z=c-WN^!vngaT zMiKGHP)KBP;=y7waqod0yIysAh^Gp@8F4)+^W%|HAA4f@v%QnF8iH>bh=OikzzfD9 zX}}~mzE0;hn0|16O#-)jYEC#J>u)JtYpuQ)}!d9 z5lC%Du+O~3kl&N(Of0%RZ2UBIEfaBC^vEgRnR3S1wiNrkvm^x%gc+XeEo9X$4Lp^t z*3@Ww#}DAs;rlxcCUNSK?od1?3l(?P!TkJ!$Y9~@5eX!`@i{Q-Om$=u7gA&bdzkR8 z_=t#goJzK5$c74|;+{fE%;>8HS;FQ>Yj|XG4^AmlkVNl-zCNNEFJkd(p!7U-7UJiLOMv4MmI z8|6o1+gq_b5q4DEg5Fv|KVWhwRSw+Nazl6qwB`caZ67+cSCuN1`Q8$0y{i;#`c{7} zb0GTGrgqPa4dd$J46e-_rGwnd%w*byP86|7FV4Tad2|}JV@FY5TZ3+_3DK4BT`tID z*f&^Rtz+@p61ttbtRW_|_W5k1V8RtipU*Bj?*_5046R0k64h-Gcxut8_= z<<(1QZ#Ge(gnnvv8f6X|9DFD!wR%G|nq3qqh)mQbuubP=v)h)cgQ+RGA{*_tFlg7Y zw%Wk88{0ClpPZ^uK%&5{y9oGv2&$-aAep`&b-q8^Rwce)!0WL?H)NgT50xc%Y(0$= z%_mWa*F{=?j2gm~$tONX|18L2p_dr2m21p!!U>d1H9-sT0F^5fCxxflo3d9p{ixjA4pcKom_@;*}FH7(hj}D z&}B#UI8jU`RZW zvG4Q2OI$83ON(bG0#l4QbE*hgkKZ$c9vA`0vyyz6lK>#}Kb^An_eA6+C6rEK&sN+h zWxjhtof&^tX4f{Cx0qFq+3hPGi@u7S10iLDXAYuvXj*tO5`dr`+dE2HJW7qCp90D5 zZ8olBJtD#w24PaMC>Qv{AZ4#7LwmU>UT!4FfOOm^yv(buj{y*N%v0(w=26 zcrTwaYNlus!!AE7fg7rOnekCM{OqGL34zq=o`s1xrzd8PC=q)l1vXEj^>S!f#{sh@ zp6;=+lJcRD`<&8-y9I?XMp-wZwrJm*3FNA?LSAHLcJ|~bAQ9idEdA_FTQ6c?;{^%; z^(28Z9s)knqOca%92g6o9wpBP+qL~{ZX@|m*qqE?HPqyH>j;3_47en;%=|d;zLhMVtP4 zg_7&5Hx|WHWZ!`WOik3FyIrwAt`wO+Q$m|wzus&?)5UTU;Z>pe^?9Dd323w1!40~; zt&JwiwIZf!e3nmPwR8`zP2G<*dX_y{`+9Eo4c&ydOAn^bdEvm|J&jjzTkC1$V+y80 zYUS(m99k0(-v^e3OD@KzIW{S1t~Pt z4k@j>mrCac&OaM(L54*3pz}eDs~`>{1K$sI!*)O0iGAf_D>2K)9!M)Mc5?@*Xkk~bFhHUZ1tTQt3yU*OzE+S&C z`I3PJD-~rpNm(RPisrJm>^f~h8to81mUYf#%rIxgClwtSv&W}VI#3nut*m{r&AKyA zq?vh_GO6a4HW<++3r5zdn`5aI9hE?6lSz)f_d{1nB^B_*~uv_EWw+;t(hQ)g~ z$|dY|c`fY}mv`Bb9z>HRMJh)`9c}ljv|@3-E?!zG$W$;UGq^X7EF219Qp*r|Kkbug z3M7egu{)3~;Kt5i_}RtMb_*q-)V7!CH7jVA=h2NVox_i=X6Wbc*?bCz>yII?7loI? zKPS0IBB!rhi_^kHti}|J9k#KrxdPpz19{?*NX17cwW9XUksP;+#!9NJ%vX@#cL=pc z3;oqiEHAB~zC{PTFfE7TrYbx=KCfIAVjN*px+mUF0=7Db^K}or+`q9^6HC4K!Bt8d zS68m1zP2fF;R6TfaPrUsZaRC0g6t4)Ub#f;mHkzm##n4p`vz&}+AaYnc{fw5Yf^E14t+NYjt( z=5*#Jxk=O}LJA^NbU}e|N)!i|FmMtL+AT`;?k2byHn#D@WT`HUVE(#P*6|P?K?>MZ zj$J9J8P-0^Q*f(P#8ofK7p;c|bc##~wRlM+rbEiMzF3vnLf5nmyTY&}YO_iPlq!)O zXEC=}B-6vUs7%s!Lvk1!TcVqx1rkUPJ`U1DWJJx?j7ar+#cSon^K{5k>3OR_no^H# zK=Lj;?m*$i!j5SyZ=l`#Rq$2XKo%@#sv+|v=+I>IDn@Ro(8;a)WX+6)eV3LQr@pMS zHcc5c4-Y~4-iB1>7nzV4m1z?h!SdH8X2c*%NP>Zr@bFqsVPpetRuoe0QT?e5Oc=qg zldEh1@tmdX=aYNvsxpV4LatCmp03--z3!nhpF@5&LXSw;cb%V zVympcp;}DFYscC&*u-fd_udb{l`-k+aREDYWKTO7j~$ad%n)^h#B83|a_mgDQe|vj z`#rfn_KAs{f-U>N_yZj<8CZzfbsm$pxG<(F;Ke>6(2& z_^dp>AN$wt#L5%ru(8~Lp@gZx=Zn~I5AD16b3^5xy)#GQwG?O2m>%)TOpj1XG}{W# zaUi7i*{9du>JRYv)ysHo=?ZQ=eggO3dpBxRQ>ZsKMFyr?wxq)WT`WIyE$b~1p~xt7 zh{k{`R*j|gE&RpzZs63yJUUZFthX(qTHEa|?L(g7wCK8df4{eL19x?vz@)zBs+TZ9 zb=tKRi)FK4W!|Y(~PGW>#J#$jl*3VttzUVSAXN>mkIN zo&yWrF^tos8KHJ9Jbqo}hQ-iM8%yQVd*u%ncTvvU#BUPo%i=jbcCHy~ok*W}R=t<_A6>s$_^(c8IX;t5XDw=1g$c_Q3KBUP zoy;bYDmgp#*<>!11Rh(3$i&7~kDhYitu)vGs~rWX#)~+;`WX83h~B84rmDi==H_cS zR_8(fij(inr9@{BQ@68!`+K8MYJx<*hQ4PIFs`eIJJwi9IK-Q0K7axp&SiR_O-{U= z=4782MCN(!AO@XXVf5v{O+TtR}>P<3qwggY*=Ti4vR;QhkHNax*ShR!SuzB}Nk z1K#df0RYxIgFwAs3{~8Er)qnFX(1HQq_-Yr|U<=y}T#^6G6s0o3P(76F_C>Ne2H+cvUo_h(Tz)gqH zxd}KBV;h5!3+|RITav9})gzr=b}uvYf8V#(nwc%j!2Jd1k$leH`|Lfl*81987{V(v z5+{_FGY{TVX;%ko|6Jtf_}*A`X1wSud?cm+EY7T1SSGmhVu)YuL9$lp?jSw{7ob~;>D@k9J=@3 z5cE;Q9E?gfNnmAGLT%*irl~qf=fo6AJh`KRp+Rr?4davT$byur!YH$f>~GTr^OVY86RD zk>PC!DO3<@hGfW2a@_Lt4Cllk<(Sdil1ed@%g~UvRm3o%_wLXE(nM!B)d(Fe3=Q;P z&5}{98XLmW{yq%#^q{vhFIrLgOzfResk47X9rL9!Cg&G1J~M}7lQTGSdJ-olXAKe1 z%Jx`bhl(ZUEo0nzkC+@(IwZ1oO`AyCw&wGjiqncEMiyEs_71U7Y}X9@12Z$SMqx5D zF^UMbo|2i0YXc3$Qr4PnCD*;oHUnKr>sZ~2)?Cs*1qtmjNj!ptO)+5htYcbS*3d9O zhF-gz8p3qf8#oVjD)AGsr*pGqkE_5HS5-x)??_hm#yiZgPGF8>Sz`^Yj^~S6qv9AtP-f zZufA7vXbse^{2;B{r*odP@O=SZE!lQ-8yBi+thPivrZdKmQ|Kh(lp6e8;d(aDOx3w z@NDaJ6&lG=KFp~?;ruJpq(G^p?Zl|qycfbtG%bTj+}1TMeys@1qxF#9Z{96qaq&3j zpFfVVr5gpU1tVz6O!#$;8m_E7j4^+|d_Ha+DoG4A&P(18eWnJznM&Ul3cgojo+7Q6 zJQxMlqEzgMZE`F1;wmDB00jNZ)UU&I#WtFPa9|yNhWgSWHKRD2qk|#r)i*V88s$i*=mDc9-sDnt>foTGp5=vjTdq$BvOLNl;k3i(6=Ne@x3JP`+W{ z^K8o#%{YD?kqc2em^dv@x<>+(M(2v}F(Zar zEvvGnA|#h^zKn&lB@8pNC{h_VzB8^>rnPf8CNg>hI68b4p6uL!Rg1@P0Si+W4+^Z_ zXW0-_J%pmzo0b_=E<7X~%9k~q=7x(V51MzJqUXBw{fVAUcx3oe)VxlNc8uVq8+_#U z-$BXM0zEBfW{un!cVz8wYcuOrv2^}Xg!LvCR$hU0f3I|tJ6IO~sS_yv;6E@tdmQO} zi^GU+)wkH|hti;WS$Ij}*h%YzrS2Jh8koD!iL>iDNVqa5xUUMa=2O^y$XIB$v@ugc z{`H%hKgBJ783(&6N{R#P%$c0+3WQppX|7j6fr9J8;b$>;{zi1B=yiSlxKRoumsB3Z zvgU4jor-pBL|~?ZeEu}0l5F|~)C#LHv-~2Qrq9}JgmEy5kt=Q_q-bL$)C)cSbYI?) zH>msbFJ7U<>{wU)0<?~kq3(Bk$~6JgNr?ZMrk>?G z*g#@EyiHfsH<)^>t)xep-1w(WTd&i>n-y@r=GQoqnMSI@VJzTh&K=EN5QIrHT9k|^ zGyNFtDZ+hd?$Jwz9ZF>>Vhze4Y%BMId*Jc7hn6c z{>)L;ae8(hv&Axs)tX$#dXuuhOh&FiM=pcD&JK+B_MoRDA3cx%eckqr^2N`2`sgX# zvu7Xfd+Go^pRSJQ#7Y7-v*(H-h}9xnzNy?zdo1>GiCrk$NV+_X4}2rj?;15`jb4+( zzvZ7BGslTSsP}MSIVjL*BH!w)DYJLNM8XvjAN4-ekY5eK4ULa_1Zg&Hme6>N$e|tZ zz0lwghwrD?hl##R<tU%H9C{xa*Sx4dh$1m>qu%mE-4QX!{D3OTbWV0UPLZFKzClOIec7=$QQ*&HO@6C|oo<*TY|a z6(-U{n5%~<=vwluMpS5}Qo=UqX@m#~aTcLWx4d+68vJTzys%w;vxs92Ax0ZzX|sBQ zMfU{^P^Vzh7N>1Sr@7}8<;z$oO`%+#6h8niz8Z7`n+zqUhZlIEWHg*vw;d>pr8(#dC+S#nDsSRzrutGQF|a|E0Rc~$(2o}06##$zXa;*CYbLrXTVkjjK0t; zWEE#AXAPKRRGeexMlU|1O8UR>?_^^u=Adv;b^})r(`+AHd9iPWsYp4vxiS}Rn%6RO zA6vOu@ASy$Tg!?BRme`45G4^}3}iBFPuek>i=$Od z9PRBc92QRkt$#rM$x>|sr&fhILS+T9bWX<8{F|16rt4OR zijt1A;qH-;<5}NB^n08LZf}E~ZDTn`)7#g_d9*h}({V@|X~ia7p;Q1&1oXJAN!7x1 z_ja7=-6n_CFn#|R#p}oXm2i4qhVz`CBU3vmZKF1=HnTq3g)?l#}8897s>)yif@u zREg@6DmU8uUAZm^PI*e8Q(nqk?{@Y=yWg6vRCSn1Ih0&hPoX|@5*f8Z?&G%V6Bube zNx!0M{99z=Yt{n88=CSy8o4pdj_kxK3MA7ED9C8@WA9vp14$;NcoxfNfRzmNSqrw<=TR^7PRY^2Oh0G|lSgkdb>Z7dfodNNFKKVoHvxRoRAGY;OyH&)iB zQjs!4XgFC&I4Y_{a$(7yj#6z(+_4-=5V6=J;Y42Bp3-~GNZ-ReMp+(G7@nKhthBhG zry2-T-Iys{jAMnJn0M0h7Sr@KO1JhTok)*Rg_22Ki5;`|!z~>H<1}F{iNhB*o|ViM zG#ZWr$peFwF?aVxd5TU7^kZ4X@UDSvxQ<@4_s}lXs;8yOb}88mYycJFM~VW9!f5FP zjy-z{spCf|QD{<*;~FWTmQ?!;aAA2AR?=oE9%j8ag+&rdM}XIQI!s9buvq-i%tqTR zXiNpD+BV_2n$Y^lDr5_~sx@tdFm%pB4~3RY81b}}nGZA2im4Ep);yZ0kE5q?CA$2R z7-}%-vxp{DYZ}O6XSaF|&E&~?Y6ufUJMl|*6ceQoeH~PlE3Gi>y+}_gDSdY?wWM>* z53HT1deQm2wQ`l-FlQ$sz2Qh}Dn(gdTIL&Cj+INtQi(oTVEKWDEvnj`@GN=I9P?|>r|F(7DkBq=rETS6{mi}l8&>yBxlC9891BP zu5bE)aKd(JMwW@tse>+b&Mp0UXWolPz{q%Su?cPp65<9 z(+HT*=+)$KZuzdX)N}?OCKfd!L{4bLBRTWZggD&>^<2#t2@S)-c>!KJj3Coyu$H*@ zN`nHLb@LO1%AbJJ8#ci{94w z3$mu)f=ah=TC4ic4aQepdr4gkjQBv5UAea*v)xmXtXC9RdS z0!f$%RH6B~qp~Po@-tnpY9YH+ATVH-ofc^rv7)tA#0y0XRYF2s6W3x{?&RaSA_`na zAnD)D3Id_@@AX_Kw5(`j1*;6YCkEsgzqcTWL=m0H#3;__En>%SY2uSIJb*Bvc$;i# z3Dh*u*mgk!8zwd@%GMfHZF1)5gV;$xa6?1Xvm781NaRgZuM51I(>tDDS+0Wv8@M#W zDk)KJ<=17F7N|rpFxNv6SFxW*2tVP-q5X6uTW1rV5I7gJMg~W-hs{Q z2cyRvJwAzl`|*9)edZKxbOr|mHXaEQNVjf0ZGGw(wcj8#JdIYIT$uN!rCee!kuWI$ zoY3N`*6O(HiDz*4llyS_rZsru)fZ#gKrebb^7!wsdnI0Z;Z}V4hj-)j%zQ*oAt09M`e$8}Z?w#u4%$zhf5#yT3M;as=q(@B1PFz5bHBovTOzt> zNzs5rCpGDdHyNbyv!M&{Qe@Ci6-v2&T3S((Wb&Mb-VCc8Po7>tu~e3y$vPClS*}Lv zvY%Q;u$0?zfk}*DK}oJ|>oFk3ql!3_(VYho0kxWNFJ^Ipb}?OjzZ_H9AgVq>Yczij z&R^1AA;~b3gqF`kdY}2`3G@~pLU+DK`@`%7gl6ul&pwycsJSgUa3St1Y{BW4sRl)1 zAw)S)a*f^9+g|eth}4ZTy~uNySV(GEz1aMq&G1Jhq_i!Z*of{0dVtPIEHQk^=83!r233zB!O&s66Dq zwYPb!dKws}v7Kv@R54;+oO0;_F%yZM3qW=0ZdE;_%}ngchwVJj`&OuQktRm?+xvwSb+n2aK67 zA-aDyZL#8(aeEC3WmeJTtS!)Q+`T1EkKaDCb3tMJM9U5?S=4tY+oSjS)g(>?h7 z*MEo^#iTOBU=RIhQF2DlLw@NwTVtnCMNXf(`tzefCc^F$&+h)H_4#R|on{|^_IIAswE-47rHk*Z+ zOIc1>dou5!U9Htp4#GBj1C7v4Q-U6f3+2XDUEpydtF$BXfYIO`21+!Y4xH)ViCujc z;#A;72`Y0Xxgh35s>f4}wqPO_pblPC5jb*qGnnr7G$D*w!Ad1%-mL;`VA>u=w~B8B z9ejLLY;rd(n?;l35^^h|MyvgpIfLeEBs9N=7X-@+ux1Ts53Jmo z$>h<~J1oJ3hqjEIN=lxHEQ?8L7)sg{(YyfKfz^sGHjUIp0zu91gKdpD5b$fdY5DIo z&w#QSvDoA)iX~t~!e6}7wUVxQw**%1!qu8eQ83ZtFTt=k>RMOIagF|y65fqKprdL~ zZ+5M&+(@VjCo*`22nL!GnH`e2Gj|y}+6cNgit$BQFIoBLrc@X8_t=8qeCfdx8$p={ zylkXa(}-8r_qpKuRno*`rc!xx*@qt{ENtUJa@eLYW#rTo)m%&% zwCxlHt8x-V+C#DMqyPY_bv*dYbC@ocalx7uvWC}=4PncQC3yJ25jmjF4r0kYXM=>* z3Uj5pN^8xhAZWv2!dATo#>;>6!Ki&sJUNg&vlpT-{%=9YX|$e|`?p>xT3$b%lh{K} z2V2#~pK9y&nV+m>ABbES+M zq>7|QT2dVt6p;zo!u%GqGAFiF3R+Pxfst*o=(kX*2vW&XCxTjHF)-uZuI2B23dnU- zDrHe@<~fLSmNAoULhE*;X%jk#hKq5Vtf@uo52hzvT>98kL-y5d~NYQ|cn zSNr2AGi!(-+=*afW$2UbBi~dWRy|iGka#K*LG&Y(n(h1YS@S$qDGxt;1m~?;j;?%8 z+FloLSWSWEpb01{`aLSzOGb`Ek}+*{z|}mjyOz<#;`hi&M%q+eS^lDGJeTW~FP<}I z=qsD&zdNN>$klo}Q;@$i{CB29{+qd9{9LJm&3fYsJ7wUmmXoH=l8;2B`VdB-4Xpsw z1mCSbmy#`7Yk8az)4PMIF*I2vmzMcuGtc6@`Cri1I3)s#aI(T1k@e+R1q98ohb8!$ zbXnSahQOFd30?Jhx>PyTDWC*~P#?Af%%T#kLXyTGIZEjp*1D z{(pY%Ptn)YjR$^lpYAX)ABzDsv}V|gitC+5t%_!i{Z3fHmUN_)#u_SlG8h$Ay6v;;lf@JxH~ zP`} zr&*euVS3pfEE(I3k>QQV7Y2~d^+0(!5k&KhZETIQ!-a)82^O4ja4(nj!8owwGHDrT z@@C}=#n}j7aB+j2pOVa^LGQgrT`^wMZ=$HLrbXqt)n!v(l@E z#cCF`Y*;s;iza$MO+7P-2#h-8Ol5Ks3@nPxO4oBR5igT4IES&7(UIcWl87jKTDn@r zy?NoJaEp%xE0sYPMLpT1O{{J+NZTi&dTYh*lw`@R*!59aGbjs3wjhE;56-Z~5w<%G z#}rM z2!^T3>M7)L$)+`UaNi+XkFA85eR7wpgXwAi|KA)oI?N1>O32D_wo zYS?|6s&q=C#4Cdi=716#A6m;w_)ifC!*HX-A3P<{da>*FxK*ozOeU)XQl5iQhF>=) zd)P{~Sm9MWTTyL@G9Lr<`TIRxrTeOZj^+%~wG>VSLs+|b8kZCwgzFzfQ-BVccBfP) z5bjeZq=V5dh}jx*BCPYP*w7+~AnaXq`1hBmQKQ4q=L*F$Kn4}taX1O?OpU%->>NZt z1+m;*38zn$#UasWiJzdZg;QHvwSy5{97uQ&-w@5VKoEE~VF-g*KFUheNH9TMUBgsz z7HJhDyJFh~8*aE?QCwKW1?QiKojWhWJwLfiw8yyYu253kna^Uw`Zdy;TD5W+C0VO+ zFH_qRM#nFggoqnKO%)A^oh_y#pr(uf#zw zkHtjgoujWlrSTAq>gQ{kTb#x?#ng%7wdyZq-w>nD<44arg6;~bu1Isbkw;Eoi2N34G zYose2t#C;PbEUF?VtH0FalQ$nw%Mv+adA#AXa{9&%wo-WR?a1Z{@yMP18kHfyX8Ps zE-%Ufn@;DYb-?;Nyvb9>TYza@e6X{D6r1F5M#EqM-MXr1c6Ov*qjjd7WdBs@J~h9F zGJ#enFq$3inwg1Y!OFWWgQAVlTqq>n~@>DGF6~%RY<$5*)o-%t-fN` zy;VJyV4~}Ela1?)Tp0lgKgyuvVB#cnrNX5DN5TO!uoX(;XEqS^b$8%zKKo9rTCOXS z`yV=pkALO|B46SZYRC=MrM0Df=k>+)6IQjl->Y2;HGd?Gkcf_{dLv&MGYD6)7OjSCUWnV2S{LGIbl)1V||*WPEbVyGX>2K%H+xp3Vo-2cq8v~HS_ zDsgCMNTDX}<#Gk|boGhD<~Lg#f)6AfE?}hR**y3IV4VmR$fFTn2bz2&kh-RRbwPq_Y(AYeVBj#6jdgR z^xhj1Oq8&{TN2E<`mgie3^b=COtevxD$$l~7#Qy(i*?oda@M2AC_$@OGcPH&Bm$1i zy(3|M=s$_KJ&f#bp=a^|#%EGbc~t zRkywdx4r!>xbxp{$C-&KY3;HK1}6+0i+Iksdi5x+r3_VrWic!(^#BEvEjNFH?s~>%PY&>cQcD468($=hu973S zb70mDkH$Iqf#x)p_)pk+Baz!MCLky+2Gu}wapT-gf;zZ*C=F)gb96GRu=B+ z8l;M(PbNfRXq8I!@QXJ{kv{y~!Z_;HMdax4&e8?TXLBf(=dmz1BSTScgE3s0VQ(@Y zpcRnwDmfZ3SV!tF2a?^uso4_j#PT<*$muf48IfyWt%h#COnE48=uFLZq&+tSn z2S9@yytsNXy=$(tM71>K8COF$$}*!x@5EdNJ+L#ZYXd0_gs7<%kA)kf;>U|qW*El# zmw|h!z*sHAvgs{B5mHTj@z39ZP3xD+_wIXWKR*8F-$toYL8{9~*BEW)l<{j%43?An zVs`C_q>EK6P;uaaA4CG8()uz%oUf5az?l}8amkDkIWZzmon^Nhwt0yJrV4URqU6Z6 z=4xcs@F12B_DY4aVc8gdzGt7&Ia4w-v5-+ZnSm|534nin3>cAA7SkY|I5Q}8EzL7EzsA&zVH1~KcQrP>^eT)$a{xnw{kZL2Z^!7; zQGEMvzYIn1ZOhh8SS-%tWjDMWYcD(>xBu(E;m+ItQ<7dj%R;G2Q}li`0B7aOA@ucR z#L2dvU4g}+D{&?_f}$^3097MEf1VP#6y!`~g%Ykz252o@JozwIly>v!+VI*91ocI5 z!-PR+n;&(Ov1u}{vnczKEI^pRvKzLL^wa%N-w--<<(NzRbOQsuxy&F6W=P6JgwRzo zq~u{?VB|WqKY}CDC?YgxvO+$HQ&iqN6rjYMNogr_4muUePR1vX(E(n;!u+hZz?3XP zyY|WRUF{2Bsa28h?4`@ol*2r9c{s(s9bL4E%P5u>sk*6QbYuw%bdeVq7jSyKfx+Q1 z3QQR+QaQ;Vqcf)~7Kp!iD|S$n%q4}+u3o{OFD}eWC1g!~Z0Q|@?VcU6rUl|E;RQWN zHxl%Vgf*pvHbZmEkq}}W8)#KUmDy2VRG#xS<@eQSgpo6&^ho(|4-5m^*F*xI|A-9* z=be!=pDv|cm+oywgBSRISQX^Zh-2zDlrp+*HW>-y_Lt$CxGdvMe}DS@w_y9$mGb?^ z_Z-HjKl4pgDM855Mi^R_K`YIfYb0_WsWK#a6Uq=2ZTe$n{bAa$cJ8C0tYHF86MLZh zmItB|*H^`ily=);BxKhUB+Rzif)3uF`|ch1legW3?W>nz>xw0K?`y8dmv8?$RUBEF zXD|uZvi|wC8h#L2LAT_9uM0s=jIu-qr>O-tjTT<)XA(O`hUc(wK4)`Vl(@BDh}O(Y zNdm_NU~9<5?J)TnB~{#UNPD!$GEM0Q)SD%QW$n8wGg5lr3$qpEItqdc;x2{jX=Q2l zQfI&NOm|M)85LSm@lI#fV0Z7eIF{?d5NoEZF1V{+hJHE}t0x~tOSO1D zo(UQiMREHy;}X= zoFCfP&!7kC{#J4;Q5?7o^M$2yZTP_qh8w2))4^t0^j294Z@va}Etar#=3y);KdrUi zkcYc^iygolZf&Gd6}zbEp;!Ar-7=b$*>FyxzZbiyTTcu_0OK#8eV6A^q-=sOx&#q( zvH1v0ImWZWRwBllG$yAx37sO-OXmelRYswshc;eEgnkfS04)M;Yn?cCfHJdr_>HO@ ze1rW%BE0BH-OHtGMIf=f;e>lYSB-x!w*>fi^DryrD%r55(wwvcs+AVHI{MK;nPid5 zVa`?xc^+D(Q7)C_Fk{UzQ7@K&qAhnd_a&6Ue1Bg*Iy-wfG)n*8apSoJFAPyfmMkrd znC$#{h278ZJwk!#3W#lcmDtWAvP>1&vYR#)hLhk)1d{oRDnKfo;D71%Rm-3^RDz@z z`An!-j4tUuY4x_OiWD=J1A~wgXmO+~WvoRFBiP4l%zFU)d-Xz(l@Yx06n<(C|C^A^ZJA)kfsEzD0Tq3 zLQ1pFp-F>Nlhe`;?%+Tpq}Mbm*$sGV=z2Vp?HA#cCN0-=gIg{1i2gz388A|srB`S` zAJ=XqZWx3qIiRy)71{C}1p^QDu3vYjJs3O+Q?}x)& zLAReC+u3keZGJTclOQ7*CL1u^cmDtWD$?;}QKfw3m05Dgagr&amN;5iR8R zl1|a0sg*D}b(|`bdBIdWR5lOv3{!CE&_h)5ZD*g8l5W9?K@+4G%&mC;DraOpZ4Va359lhbA%)mlr$-3lFf+GO>J+^6R#Fc!g(;oSlS9L`Mipfs-ohp*8_ znnW22)Rs4{YYsiR2974%>Hf6QEC_UV8Zf~y0ms!Cc@0n2XduV1tbI|HmDUK`gtC@d zf}3Em@4j#6S4zYdbxShgGHKdbum@s-TxAb*M)5RSCYaa^KCq62zE(DlWL$zs{zJ?A z2UxYNA7A?Hd!;%pmuvXIC;k~Hr)H4qq@-X~AHtkVhiGt?g{(?=a_H(wq+mQfJJM^@ z%m%p)+cL1%sPAq4ar7KO8G8oXy0jWju8{zWL@W`u@vZ{%xX{3P5s_UH$)CvJQ&aLO zQ%>W+_zbSux>iWLTbGaEkwa%NTQ23i?qJE4aF_vnil1w>*?Bv7Df~1 zIC!+R;FM`;tKaC$f+y@i*;uy;pd|3M><$MrZih7t&q&l5jBuwz!DNWO*X7e*bDGHH zxgEypy86tqZjoKxU9^Wh{W(}DCAR`cR=ph0Or>SDb(nelq&&BUJiMx@;ieH z!s@(S&!ptOo~C#*HO?v8x77-&enyCmJ|!W>`bR2jL*xQC=)$9uXi+e6DVQ|bX{SyJ zB%@H}j4&SV)*;jeWrfJ+38wuOCKMQL3rv#eh;QY@3LcS`RzfvI#oBgE1YMY!!L`@F z6k921ES#CerPp4GB|9#_vi^PycMsy_H{FQAEnCsq+lO!cuP@^KtFFP7*ItKv@4iQP zDvHaCFiXEGE9X%k?W3Wk12~gfhH7q@0tQ`IUb{|!ECmxzqS?NfE;VN9hYOV8_}yuY z^iZ|5I7R2k0z$9J1NfUhK^G;%QC64CW&tE2nx<;8R-qcdj0ncrro*qCv#eDE?ukj#hRIR_`5nT&^GaRGA+MF~`$g-&TdF@lTv zA;Jlz>r*UE3i>OZ;n_+->t#l*1Rt~Wi-LIRD0B!}P9VarCS>w|GgrkAwM0yof(ZAT zwUJRMzfW0)xIA9BiDIH;Xl5oRm>{l9L_kr3i61kzjVDM{kr=wWcIVR~mRy~1TS{7T zDq%_`&V+%1|0yH&ZD6n#%@>4G63})q8H5asprPDT-#5wu)(9$WWBmq^&wMPP$pPfA z{^Gq@Hl_#efA)p%;=v~lAl>a^XjLyISSo@}$DKeG>zjqjFvTGi8YBNS`7?$q^Jied zftVL5n7@@&Y<%AxqspohR<%K>D@CiCY!!X&fncRpR8p1KwAE6ct-kg1_?~1kohTG~ z1&P%PJ*q}3xM~z)ICyzH$Z* z1uG3*PnlTi2|!1jDP1}|=|wtfb>!$QXmoZ-b&^r~(6)Cu&-doYI9f-Zq8LQioc>p? zQ4nfrP=cL-hLN~vf?(*xUPLMhQnGf~P=*WQL^DJfr3$q1_*9#y2-7Fhbk5DqOrcC? z)Xl&1X58?;cOvIxaQol?3s$XNiyPkme$1XcgFQdrg%d|l;`5*VUpTP$S#{Erb zyoVk$d64e$V#8zgR-M&0p#w@&mtYpb!)+3}>47-7Y3OTIvIT*0qUWkYnL%V2X-?Vk zFCU!JgJ#$nb2qAUXjNxHOHB?+&dRmSNP!|8ZFVLLG9oFh4W0PI1kqN|);bOo1nIH4 ze1~M+&6e(Kax@hW5!z&P^OLBSi_$*pk&;=Vv@nlqaTcRXhLOwVaP-(o2~%A?og#Z* zuGG*?K_r{@De&aPiguw`ld8crfs$X35BXZ9g5GY5)N_-9#pi*eLys>P=WwJ_#nz4I zp^Jh_mEnaglgaRPlvW*87oJ5CX|_7I+!z`hNOoy8ntUs|trb5LC@qy_F&ise%N&m^ zXO)Wi$SO8Ta$P$bZc3|?Q9UZ3H~n}-*;87yv4VK9%f@I%8bsPx!f+e{Gppbkn9}P9 zuI96_!U5lFO@uVrnCKZHF`EbtFC);cC0cj4-Fyws-@00U`qO*&;P$&8L8>c-fmOqB z@=7X4Elj7PmW_fFS4aFmCTJ=RT5t_$!CK+k04;}_A1f_~k%e&pkNF&q4zzv{wky@C z*ee1?Ld6CUu{$=|D{DA$FlNC=?S`nE7!2uUm@TvU`Mrs^1|=-_?LCBTRFPf2X%#ju z8NxN2#&G)szcd6}zunDKC3Kw3ft){)-KFgdLA5zB#z zm6hD<7O#CC>@wBd=|gXJ1Z&(8q?$!;%+j?8q-t!M*-k3OHH3!Gb1}G(?!a8COV$^U zj@h$@_i-wa@eA@D4(%^sagp{e1u>Il-vH$74cyQN!80;U$hE3~n^4Kw) zniz*-9+}GH{s$h#{SQ5Xj?NyqtY;|vEe(d^_b{`-Gwc?Ybx$KRw~O8nB?1FmQKx&s zxp0lH^Bg^H-i&n8VJHycgQYAg`7L<)h6cQ6cEMe_`IVmtf<}fjbmmX6t`m#Ktpfu0)FtjQPbGEnl9=#u5<9Y-G#a zHq`=uHXTen^SKiGT9K%uuArL2je)Sl<$47{RGC=uPV3dJu_a*}m@|@839XH)nFGlT z$rXMFx3hAY4s>_+A(!pcAPocU=GH7%Ax|7UDEeL5bP8KmjNwPWd|1Q86Ehcm9o=8) zBV^}|Q3~Rygqr&cTUqDQ={lEYlsbn`%Gs`}EO(?$yl9_OMYYr`VRer6f-96PYJ!}# z9ncAtCiZfK;X=aUp?8U{3k9JXRhD&M1B9}H@l>Pz*h#m4>%u{-ntPNInkmD?ha_o{ zj1a5ehMXwpFg11|rq*7H#dOM8bs4oP$5!A#p-F%5JBa4Qvvhgc^s0rL&stoX$7KX+ z62TLZTz_;9CUl-fK>)fV;3ZmxnrNA@&`ubqsxW4L#}-Fc(ZIveqepT7gZJX-@sku( zdr@u#*nj9CN^I&z6-=$gnqnzQ=D0J`lvBYKDHRFHO8eh%qPt>wKb8!o;WcOA)fZXh z3PHypLm#kM4N>HE&J`j3opbdyD$=n(-Ec77H-K#&sfPH^YZ)hB22Ay;)_xH=1tW73 z1Xw#Umm`fXtT6<=B;&pf_a2Z`k;-hKp5-U zZlrTv!pzNP=ulG`+2AQRE0(;D@eXl$GiGy9JA~g`v)MF;K`m6uMPZHRQVNyIBBp01 zFf%tUEuu9mG}&!ruppTZ>x9ir&f_B={2hGqzr9rp1yZ{6wXc5%$4^dB)zVAvB`rie z*1lrz0ueI|97BR?QD#)6n^wSvH5cMl*KCp3_~y6o!1LoiHm+MIi1Qay=ucz^;` zRle`U+Mb~<@mod=GYqW*qR}=}W?&dPk$I6G_8O^pUxO7yT0<%mU-|;PfJ2MsGAE&y zM4$kSj7TE1D1e+fbBf;o8mwHk0?!^iVOm^NL74%3&S}61HGNNeCIYv5Z-+2HR$Qlmb$m!y&GG$uScrKLy3}* zZn__1BP&s1Pxi%16f}a+jm@M&KgtYkCC%+D{@I4!F0^Qk9-Nqp^`rQJVmnyocPL|` z#!ibobfy1gsghNO`YR|QKWLMD*ULS2y0b=N0*{tpsR@F@$)G^OU=6NB8?>2a#=vlK zN>UoNvJh1l>7M?>f8B+Tyy0eaQPudCmtBj$|IwW$b!cpP zf4zG6i?qpT?PX}6^!QUKxCK;OAyxE6S?}DDkh(#4wlGZ@cP5)b2PF|*s)#j%xdr6&RlvNv1{GJux2<4whC*Fw;2k zVGpG3>lj{fO9Pyg>9|BT+*;F%Vb>}|By+++u^H+LE#KWmM}CK_uR6o{X}xx52lPmR zA48ch|889gxiPBN(OsfYY3ZJNBgC;|r?Glfp3YbY-OWj;*@r1`G;qA{e3X>-$MiyN zU(A7InzmKL@Z*k^^SEd1B9Y6Hiz4R8H0cKz}&F23Lrlk6llnqn$!xvyFi`DNE5JoyL#KMk3m#hOu6Y@~E-YCg@p-#V_xgVhr#n{WD~AVh>lK&cd%t*q)@P}$z4ZTlu>S_Z17}%A2QTvcD1(+3jQW+j8A4v= zs@v&6k*fQ{&z;2BvLzTC>DHpB+;naFIDUK_gF_?eSlUI`vW_l)5vk&1lBkvkFU15^ z@=Xdnz4Wr{OQ*4M`Z1*I$5H19U)X z@!H#d7rS;nf%RLimMUSOFNc?2vx5%9UhI4JS&WY#5@_bw*iu}6>7{hQ*YM<%Pvg+B zaY5Wz-YU<2R#WwJ?PVM2a}=;||4A$?l+fLqlkBNl4=A&4VPvR}en*EKey7K0P+VM~ ztTiQ{pP`S1T(^7|T$!*_o60KW6!BN*z=;kwOhQK1j@tsno4Hfa{OTyi0L={nr^ z$X@Jy?l2_vp*`@rt1ibsQMCg6|En^7!&)REZy68W&qYYFUMA<_ zs+{fM7~#Jt&}VX43=9n#0E91`3r1n3(;1A8jhbp&$!w8>e;Or>)}BCpp6+Y!m2^1f zv8pzK^|Oy5)jSU0)kb^DR*IpXFgl^$CkuV+adOR0Oj1Sbx7wS6=KPm=3Ab^^=Eu={ z;yE0y7v%!`fv)@+d=sh8v_fMYVcKX3?YyYJc@_`PDi~EGLm?804*h_rquVGt3_!-OTY%y}IavZbrS@0}{RI(MQdvG2%q(Hf+E zPZj>ya4!W4Pui1R{xsIiKZaH+i!%c&!N*O7esMKym-AIbdwzoxs$4ehsy==;@}xY zih@fzQ%9k*Dv(EJ*_Nv%lq(D3#3zu$P!k$C@-qWiiGpa_WJ4M$RO5q=Zg@81&;?mU z(5Og@CrhD}B~n^c@iePkmDB`M>ly#_sfj7v_FK2$ufF&xZJ0B=h~MHsa?g*^zx)zf zTp^}r7jWLTF?``q-zbmyAAeE7&vw;>34GqBRruytKOx_L^Y8vKp4)#wh+^WEKo!DA zKJrFfcjZoW6|(ZZC!U_i7ruNK#-}TEsAuGTz2%mRaN8|A&|lCktY;2Y@wI=u50CFT zDhq4-wjq4r4{o7sZ32%yz7MzFdO5~M^Z3gz{1l56gx~w_oAJ}T9>ufIp1@n)@-mDJ zcH-#KllaCreu9S|e-4F?Zo0NDbW+9h`d43rYp%UO@cYjl8pnU#@emFiJSB(z>u_7nxD)ovb4xmNBw)@72 z$+o9Xou0ubzVtN;u2sx+b+yUlN^T8Rg(nnJ z5~9@wxE09uoz+GhtorCsd(Z1%jjJ!%g8LueE#IrKPXz-(nCULa@NNa>M$3kZhx;h6$CyT3WmPu(gOTsiSQ!>R8J)D#|N*^D# zEo?K)UAy<<)D17glKwv2blFaP6Z zg=MTNJwtbb5}foHmR1j;P&jp)vWFRV{nYzFZ%pFt&>2|2v6g~LEuEAu*h_VTC{<$Eb} zs1)hYsdedFl)veq^_xPPiRX7ZeYdo2_VD zXeHsMkik-}pFqrLmarvp@}YV2E3U@)-7?dYIGQvUeGU|;!;)B z=CEP}ST~|D^7`#`|8^js(L?WLgUji6{XRbUM_*}SwwyNZB{(Tn?7pn}{5own!-9IkTw{6{lS6z8Ae){Mh zeDlY5VaM9_c-Nb5#uNLW!BhM8reE**NaAIOsgatW~xcS#cK&ued*mpdK5|SOl+ixe zzHT*od;4(Y_z9euou}XBh@fI;PcOQ=21NFRGxBP+rsYnWi)d69MGb-vpardK7BuC*%d-SMcuwDAV0%;-?mLId=p&_SSRk@+D zhIKUh$xApWs>omd>_0&pdNbmeC5%z@nd`_kX@%B<#22uLONV9I9l_E3MO6Ly7+8E9 zeq}=1>VSd1yGFx`lH6Qa!HUw86hNM%_eW=ss?ho3+3;ZW0l6#C5G2w<_evaJaXF?6 zeRQpLJE;{ps+yse@fS%w4+=-v_@B{OM8&GHU06OkF=oR}Z zitt>TKcg4H^BK*BtAXJ4Pqn?4-pxSBVuQinKPDL4%V6o#<8udw#}pfP|4}n z;tR%5Qw{n^l~+&{q=ufa4>^Ue)4s={DRH6@%oL}^~eF3v4I4@iG@Xc@o&FJ zpQ8s4KY9wUzjclLtTUg%kM4d9|M1Uu;L=Mr;J<(9HWYHe6+1WKXAeDxm%n636iB}M zk9Xt2habV4-gFgSdEM3c!*^eg=Z{U|d;hTu%f@mT92mi|6Vtf&u_HK02j-h!yIlfF zg%0|!f9GyGb6a@ZZP(+sZ(fbbGYx$Ahr95;w_hrOWZ(WH_|bnpg5jYKs`#$ORTuQ& zz;#>jkKcZTD%KFaojUk-bZSPgmFD-{G&-C-Mn@YWF=ML|?&)@q_TZ+m%cazxDpj!W z@L{y76`a3p6x&7zFu!&MzTR>$K$XWuO9!xJ{O+Y7LaiC2UzbhWEYvGVDIR z1|R?OzoA6$X*mU^PrdE680zZ4$N%Yjc=q^7boC9Pmp)4&-z8Zy3w>7WWnrt=t5k(i zWlM`R;uD3DnUXSA(e0(c&z{(Wci;3XN*L03<<1MEU=pvtpbcrMxkB1aRLrxC#vB7z zbdOrkIs=vX{7ho?wUi9ERV7&qW74W@RMU$`g1N2h2X&S&yYoV`EUQ^%6r58sYwp6K z+*aJ#MHO?Zft%9`D<>bJL~ub`Im%cis#u&3sels0ps0Oukgygv9TTn;9iR3~=L#HK zc_~g5dQoF#3#HX}od_hQ1gOHDmVWwZeWwp&=*%7zf_bo1s?TLOL;71R^{$UhCY+151V$a01&tWF`OOiC@JhZi*Qx3 zuMff)^_4V1i`t6`1ek`cD3hbHtv*^hWr|>fGV11 zEKcc&55^4s=+mFZ`~Ki}ars3T;}1XdPW=C^OhJKp&QnPUtOcHx3et1;ABFo#rti*~+5 z2In)U*{_rW-Nr!-^>ztr%>KFO7sv3e9~?n9J+EF0swQ$d)d+|5l{2E?)`6c+! zn|=#h#`?qyVyarjiv9sq8X>-O=g;xhcdve7qHWB|m~-<7x7>gm&)jmYLx2;0g-zok9rZnO&{Ik|$blks;in zfkG!;H3OprkPIZY=gqCLU3>OQAlbHh1$w)?F*83OS@$Y|L$fV(2Kj8K_9iiUUaXbn zcx;QEPUySEI(pXlOn4!_b`)hOKLZk2{Ez?EXfv(-MIlG30mrfMZ}%Kl-e>t8iF}Ad z0mS%@7$dRSj&(TNe*vg?rnQ^KkDX1pbr2YvdIDi(lFk|fmz5ROF3O87+%jGE2w zOIO0B@0W7RFuvky9O_z%MPCOI+x<(K)|4iZ0)s4zf8rVRoqQVI!IY>y7F!wdC`lQ3 zaUjrkjmeVdx~97#2XrX2W<8ikR2|r+Vj#-IbYDCz$-}(6QmrJ(vm-EZ7Q6Ixg=OSa ztoMqxJkaP(6{;6)H)?@im6(-;kpz-85$434lkYrTps^(b7%XJ5s;d)2gKOce+(f^t zCIArDRvP(rKqHTMrop0bqV>t%>r844J)=2eMZN-K6YNOaE07AslZ+@dj8YCB zENh}$4%#x6O|2Rtf#M5ull6d@7|8M_tgpq02(6ephw9?E_#L<&2a=S`Tf`^cWTA`Y z7C!vZ&*Qt_`ku&nb1+G#9enOHAH!>Z=VP>ySWcd$-ct#|KhG#{YK^C8NL;4_f=PMi zP_vtihtGAYVCEL*F*`dg!6Z$aVW>y9gm}Te^KBQyJBudJQ^?R^l-AxPTBHQylQnd7 z52CX>gX-j5#NsY3@Fve-uzwKQOi55}(h6%SY)KFP{;R)F&to01MLKwM z^8K;lEOMD{Vb#lKqX3?zjJi+=>0s24x%9%-_~*a=fV4a59M!M4v{N4U&|~{?_0ILU za7#Zf{On6GF?$uBc={Cn>u1Ms;JI;hb!$nN-FwS;?9o{CpY4m-}t^%A@YUe}6~ zq?9JRc6R71%<}7uz--itC{Zvh73VZq#@8}GuJv)T+@W)T1C*JZNDwud^GsnKPW0`B zo1^#4LaRBhaq_q;w*|LQqS-g~Flu6OmC>^cI1y7fFq(nNfFJa=qceflN7d=U(SaR! zu5&5Mv1*SNcH~x{qXUXZuh~uC8#?nW`cLmcwmD1RtO+}uecvZ%r!ZZbk+xWq2jz}y z%+~zKtSnk9yhA-*y%^uCh~X}(%1Zkbu`nsC0`Tuo8y8uK1e9wu4K?q?H5@Ffa_uP{ zyyUZlNmi;#REwHW=_d?AZM(_~(-Qa!o{zPoz1X~T7=v^+bQ$o&6N{AiZ`_9b#8dDq zGn7E5G^*C9aWF5+r(iwMI)%){BbZF(F+02zC8d#FQki%{mP-t-;6%Pk!jcZn1u{lS zUf_z%bdWI@%S)hF!oWDMd0IN!abjqxxloar6SSd#qa&6^E-*62H8dY(hD98)DRO6+ zqhL`w6i`x*W(qU$4-`!1=4S-~R-n48S!)S2m=B&*svr<&CJ8cj{s2XFeVCvoM@3$SzN2K?dse+Qrc+wTjUH8zMc>YuGZGky~Rpds#tvFEo8 z^ZNQFdux?TR81_!;GpKWFf6uQs+ugx#rOX2&vD}T1ajSZR7y?CoE^&YL#kLR;!k1I zky5QJ4FAB~yqIyzUf{;F!$~GDj+C}G)19Zx{=lK58%#w4%4Ce9lY+Ao!GQ?IfnaExZ&!R z^7Ym49maQl_&5bd3JBbyq)eH0nyfmWlFUx&L2Jgry49o6yZ_?1eu&-AA4LxjwRwDz zp=5$pT#d?eeC}J+6hv33NL6d5vzL|^ty2!H^x1fwS17|iZ`}sG{<>>$QT{i8$@1D~ zRw-|Ud;uT3={l^;XYla80}@Ow*|HAztyqGMYgWj^51uJvU~D}F%5LEzF_5IZI6+m( z0%~-CG$oJ(5k@K!@{6Fic%g+ubVmX_e(;cx=a@x*_4(Ve_jDb39_*(&q_X3xf?F7i zW$q%>wWDdIfUNmfmbV=m(WfYoDD9pnx?xUAsu*c8dl_vf%{IsB9ttp{@cXH1sV`um zG>vj`TAWJxkmc+;(wIVfh2=*Q{ehefi`msU(tkB33IoDQAEcknxY|^)N{9K+-EDY< zQ%B+SGm<`}rGvdT?UoA{-dmj2S-EsS0JGLPG^Xoz@y>(K&(t z1{j^DO8>;;=xI&E_Zku?9GGx}_rTjywA zi=E$?R$7Z*uiPy2Fig%+;HPv2<3$2)NT38Estv_cLhO~G^9KW~5Ajo7v8 z0X(xXiQj(n8_-$k(S!bpDV#ic42|$LRHbAvHJ8WVeCH%OC=-0^t(V}Ui&o&l$DhY+ z@r1}Bv}yjB@|>j_ERetv4+aWoJdbk1NZpn*uPOzN@$m^Q*+r~b zH-Jm&vlQniap)M8+dShk_n$d4MOypwRPoen4f)<3_Z-9*zj7zK3LdtsrL1Of0EbUk zX`^JZcI{@InQP!PUwQ-`nFr~=3%Ks;b=a|K3toBkQrvOxenIRs13v5msy2P9iX0jE za*H*P^;Jizqk$#EeI^@MSmbh=u20cTi=bf2*L`5tO0#Q42%6mIPvpZXCP~7RwohoT z1$(crTQZ8zzw1rtrmAT7z4v3wn$^DDrD{ModXVpk<0!&y7E8XC^BL2VV~cFo<(zIuqL&hvpDr zyI9VOjkipSlBVZQO<@~VMB7#^M{nO0UC)LPW<4E81{cfRvvvEiT92bEA@xs2Eie9~Z#LTKUp`V(B7B1qOwsoie3bgs~nlI(k-_ zo4X>6bd-I=Gcx15JLcE$%>D`NdFch%cFC)dy8nl8xZ9FR>9t|Jg&M3aN55~dbpp<* zd+A>Nf4sd1oE>L*F8<8S>1}({7HL<~s#}sJ7g@$Owy_MRV@l{Pfsk-Rav`1MCZs@a z!k;8JgcL6L!UYV*7=yt@wsDt>WHn1xS@qTSJ$rhcnft!)_syKMl1#`i$$!bltFve4 zO!>aIKJW8f4QJUZDR~8;*F)|+Q6Uu{=4zPkAE5a z51zomqbIOobQEuX{U)p${W4`55e)QoG?j!IHeHnoQtxZamOr-rK0@Cvc`A)u-K%R4!IY6|12$* zqhyHH(2#2?OTmz9FS<1?j!=Sl{F!68?KM|p^~w%>>XWx)=VLG7wYLr9)@zqzF7M#i zfA2n2&X@e@-O*KD_~eIf#KVt2EBEAGZ|=csZd!-k`_jl|)A-mYuEXZhE`0rm$M7%T z_z6Dq$+zN~^@I4?bF=uu32q{X{#I7)3woYY%9_Hb#9M*qj6~GS)Or<{p0MCRfrFeVi!+5HKGh4?yElxk z6h|_iE#QCO{}}$}hHL+Q(-)kj(7o;LUm~TiWTk+&U3;a{S2<1*x=j(h%j^rYXbIN9 z$l;m!9Lj>0G|Wf^u1GA&L_iLlAcl+|EuRtkx#`N?8$JstgMYw5!ZuAlbo-O-W5i~M?UqH$n zS|F`PPox?UCld~~p{tz{YrR5%n$6B3&v1M?Clx?8>i9BMLDyIhM3D~0We_gps5wLV zFMTI$&P-IXWFZs-6R&g#Ucm4NDcuW2e5BmvNcolxdc9DgEJO8GBD5c?K}8VB^i0e) z9Hq}$J~xiOiGxU$$>H-5v2fveLx?tI;r;{xVSM3>cjC7{{XVQ;u?*`*`|<4F1NiJ8e-6L-8z09g z8L1L;V+WE##Kzn6z>!nrES|te-~VQ;T)srE=czsW@W+4l4PZtnM&#Ql^JeeGM%3jtvBbd{r7NF^!eQjg(FVhUzmB7IFWuWGmk;w`P%vwsd> z{r3I%$S>WBH{9AM-`>|<_dJ7{$#KMDAsjh2f$dkU#mHa_e&Zu8$mQm6$2T9v=FNjx zwRAcD*I(S||6R6fkOa*aA!fE&qUS%0s8d~Vb-bW)s#sUkrogl0)J7uNbK(V*fU zvlI^o!t}mKm`OvUh%$IXJ9nNlnP14r&;6-5-hSi1&yifJ^kpQHWLQclkX|Wg&K{Gj zM6B&1eU%2+oX3F0lkT0R^H_Oi4-WKiMzJM{iehq+ij6}KDMhAfbY=pBlP@Aso{)Rz z7?#hDu4dxs-Z4lbKZC(o2TZa9wBXSiHjpF--V$;tSg;W@O7xjb*czgsfSi}i&nb3_ z3ayW`v`6lFE{L-;(`pn|8*vGcwCuP7TzInAwz^Dcl1iV6y+I{xHqEzqvAKbU=-SUs zusy~;4{{D|Pbd(MTUiWLr_htB;lLC*x{kHDc2fm`1JA&sz}5+cG<}nzQ;~iZIoxjg zo64zum@$$P^!*#NupTVX6whxOb^~wD&^3vcEsd-&wBVH~1BXf|M{Rz2ud4_Q)trfe zOU*hNzGjA@(r}ONo;@TT&(wp870Ob$0WVY_tlnlRT63pf6po`H>te?{#PbUm7CD(P zo$LHu4xjwgUx;W`p`1fJ7E}CFoOX%tD09Vu^;a&`p*FAi69RK~( zUm>FioIQ6O<7ehDaefRtA9_d{pi{FBMn*5g^UsZ8$6G&yVlF|(!NBepPobkDjN`{n zVC+O1R=5-KL_0kQ3$y7c{^Iiw;K!SeqJOAYuRtmuzcBoII-g=3wZF6LwNGJDYTPQ zx9GV%d*BqDh=T*sFRHYGUh>D-?x4B9QaweOX zNOxu-C(o`x8EZP9!+e3~7hGW}X3)pvYz}Qbw8<$;IrQ=oe1GSoc-zgd##Ji^|6@bq zj_N}gk%-V?e0cjZvk-2H z0!+=^(yBr0YVuaqBWVsG{fxFLu=E(Cg+|0xvu9qNm_LI{ca;ir_mNYfjDxe4WNMIt zlXi*`W{_T(q&2#r93<0txM9jZvRIehO@AN7bmuCWte&THn=}+dEst+)$ezK*^N%4! z(Qkzz_Zm9w>XMd6rfK-XQEH4m?&L7CuvaQ1`-9iOO@*cT?r2Hg(S>O&JNG=2#WCTg zs(Umee%7OgJL|03(b|P2UCYomV$pl2Do~_@KtHEJ3haj(WKOAz*x)--Sm7^8Qb(GQnZc!MOY2gyosRE z7^-YKQUwVF9K^#62|k9->g7mO`f+?Zh(OzBT(ges_kl;ru9lS$JWKs@D8fO3qu7U+ zl+MB!dkV%4Z~OZ!%Ot;IX5vH6JbegTQ(&7qD+x9lKhO$@;z(*TSm^0*SXRK-V06tU zz2gf$(=@%|#DE`Rvp3XNPsZpp zi^ylDaQ5s0Oi!OiU-uwoBj+%E?qxb@a$N0njG>L<$kLe^q>@X~-?x&SM}^MmERK#H z#ZdnUdir{WTd6hGrgXi730lwrl$l+CZe4^GBO_sn%EJ`t)JkQQY0<+E#9I(gsR?Ug z*ez91&J|!%lpbU16d4Mui!MDY)iGj94I)uwMhm}0*;<+9Q~fWyIvI^vBCghiO`n;alX5{K5#!n^xCdOS zvx94;>_GL(id8Zyb`jA~tMq7`WCljGAW%~_48QyKTd{5Ruo%w|e*P<{a*bU89I7O# zv`JiP2&RHOlkOm0b3`yHVwcfo)&z+&=W@mqr;S`iTc|E2g&8Lz{e@sPjpe;PSR1nN zo{zp;Wf_k1A9E~9yH}1gcJeg7_0Xddftr+MJagy>atnEMlf%Ax<7E`#S$Ocp-J++n zjMl)-TdzQdoWxI_dltD;4Q(AuG0?vTEvZ&|UM1uU3$o5@9?uv(<8OZIcDjc_JbUyk zzWBo@7fJEDu7O2L7>481qI+`4o`kW;4ZF>f28{^VBg<@dG9EpL#~AIERb*n`2nr0D z(PUxDR=AYRa~Q0{HI&5+&B*GBI|cMKJ$z9iHMQyb3&W@SZjwM2B#N!( zRzmTp&$%QZfPkXv$Wl~lveH{IEyFmrpf9Q_{f#Dt+uxyxm4_eO+q#j>&0>xgI^Pf$ zC34+}G~u}C5oOeosiu>{F?#%NNlBiXVJVw+cJ#{pOFl~xaI}LAG8xBEK&I{Ev9JvO zDH6vLqO`G=4GoL-3B!(g(34q=lL_VM7%9_|OrnL$W43bNqKVBA$6AFxCx}E#hdfB0 zK+P43lySvWnoOHF-kwy=I8Pyvt|ygfSNBx(*>w|^3_Uj$Lx>Caj&j`myi}AWFj88K zVfrjKJRx4#CP$KV4HWrD!(Ae7#Zg?0i^itw3qo-Nu!W*HW-8#h3cAf=Ou`Y>W8<m*1p-31?P_SEtBn{VkjOHywCGH= zB6aXED`L{;o-I`9oNKg3QqoT!cwMsuhZr~FquQXp2vIs)2OUmA4zNU2wLnQqjUiF*j8S&nPx~W^qBTXCMpaB_O;AB+ z$#K*$O9wyQz7kxyTvi;%)w53_S~-K7z=4A@nt-NW4`4J@D=iQ4q}Y4Cpq8mu)@KhQ zI99>op{+=@wqy0=UbJNn(f#GpV9@V4n3{e?DWL+nXdenIufuF|2`G+2JFV@ua0<0D zvmI2F%Bfoyq@)*M3;Pu#ObrFcu95-CQg%PfLVF@z5f)B62$!8AV03*lO~V(i^D8z@ zS)z<>_!EeJh_9rW4Uav|8&)+JL53Asu5!V`GGf|NiDA7bCf#J#ktjVUo*iSV{m$xn z#7e`+k*x?j7G6%1{p`35+t!f-JhTf2ljjB$VXR_^P-;s-%E$tZTGOSP@J5rauiNyHoN>Va&ePd-9${EQ3q^ab5IZ_i? zo~5+VMT@#rq$rbRLn##vXgvv@IOUm!SPP1!yhPQh)^3WJBC=?hrXa#YUh#G~Mp&BI z46$^CxG;-4-~AA|m~vTUcZ*C5o+slFO36SSH&UfwV1-dE?ITm2lG5K|nP<-Jnrx>XGSzC>gr48x%iK&0j?mQ&IjXX!_E{%|VB) zTfCexgv`Y}vm)3fb4zL}J3xlUis!_U@T%edmt`ngelF7qgqfT|SZk|T7JS3Tn~=+F zA%=;2Ye|?Hs*3$J)=CZ|A?lR^pQamUg%g7Ob2WWFKVOzfPB{Ms6p3Cxh0bxy<(u%W zHLLLbM;^zWyLMyV&LP&8@&(51nk`PmO0}c4r3XVpt0|l9qHLsse12MpPx+Z}Q{J2K zHnj})T)u=AjDCtmlvJ;F5~7f5yFTT-l1%nEnv}_Hl(vip#KxOuHyRGd^@N)|C0oZ` zJVT;08|Zy6Wo%o~Dsj`kPDVFa3kz2t3)eB(Vn?cz3{^(a@$=jUg%Yi`bJ&`G7SqC( z8^HDR&!D4pgn|jK&4*-l)ny)C1+|WVa>MkjDHE^Lc~&@+uJA;u3OXnybnSqGu_BL= z;!B91qTiDs*B zUqpZd82TJ$_h3X<&{FK&(7l&3H{$6Y&X-D z_fK#^GsL6c)YS5tV+^_~B-y4Oa#Re~j@IcB`d!2fVlnTKbY< zzOI6d2_lXWUZOAIIKqR((XJH=iTWXgfHF$P$(5!yqreKqlG4~pg!=Xj~+ z+#(?)4IgagQxaohHfwcw;~{3#WW>c$?In#YEW&X%kXX(uw2ho5?G{f~ z8j7Hm_i;^uW?b5fJZll5J+a$HFd|i3!)oDSPBS$HayKjWl(9 zOj?~BMK#`w;P4jNU&XN(=y2l{ao6=)wY<m|(CH6`BWsn|9F8Vo%;Z}{vSZd3T~Tp){S``cEU=rS(fkcns}-ZYw+4p?s_ zWU7&YP>}mMKs*{0n_^n@%q$o2N(pTg7{_ci4i=~zDB2bdWymget-~fp%N-+!LeCIJ zV9}VZDFR21M-rwbD7CtxmoKY&q8{-#6jzQo17Vl+G$NFUvd}t+cvWLS(bQA#9woPu zs#F3|e)Y52skufIG)|}Q4upbR8ee1(a;1U*#u*7=SISy}%>Rb*ic?w)%Z|WQjxUEdX%FORs|`0+Oif5+tCxjR>N^Q zb*-(VBoJ)PFDK{D(Q~b)^>Y4;zXa9jAlcfAPrT!;_|_kO8khC=2v1WWkigK=br>FA zC1t?b+4HDS*6JG^FPqTtddmyESdCE=wsFnAt_fLry;_=?f`b|v7Sn2K(K9mNp!(`i9Otr3sJG>2Uw37ltQ{;Jt+TC>2 zESwf}t5arOWn-yagTDto?h_tnE}HGAuDljgd-5uoEe;k}9pwm=&eSKde}X9knJnFSAS~>gNF^Ox-7e z4#*v&sS;#3O^(G`oFy8HW6gO*Q34g1GT@Q-7_)dYES;hG!H`m6gk)GY+Kd?t6wabI zpQmSHh_Y-)#6W@qf+N`&4s~xvWpJ}pAQ(E%mNB49kO4(b27l_@&m{CDrk7T^i|mTo zgkRSP3s-{=eTwFQYzkFcX!2ifqj}Z{h)h4?l?~yk)^M>!)bXlX9nci25TR75)zQ`( zlH+G7mE=vu1aC6_&s;Ymq7@NIU@jF@D;2~NT{2Wupj4ZsfO%uF!Irrb?bulBqd@7Y z(w+~|-#Zc!*D1AQdV9-IFjb~&w#7D8qyfoBzQlbS)*6qPGS|RfpI2p_((z)UDwFw~ zUGepCb|lJsToS0M=#8(7XLlI6WIIZkk11PNFo-{gY594`C?W%f0R+N|rdv5t(^ok3 zG8T`;{32?Lrf2N`f7j5pT+cu_O_U!OrZ=W$I&d$@P3((23BRvyQdXHH6OpV4D$RAg znWVZ56^f2d(`Rwt#VeZ`x_+-F#LXl9{18S8(tvhG5vZ$SyStCQRd+_gFV^iODU_Ess`JLl=X8~^g#Ba zAxg#QV5_n+YoNPc@l@Faxd$9nMfy=0x&gC^rLY-M&ZG%#E0JOX`7^u7>dy#YpB{c4)YtA);H*kYG#K9yJrHqZVM$QpfU2UZ%_%gtgpo%}2*u!3ZLe76gJ(5uXzI z)ks9jS}c=l())|q1(9N{mR&@m{bcmxs5xqJhe8&z1x7N(#c*-+mW8-VlvRvROma#A zw`O2|?i}jHNjX@ zz)-!PdH@ki^+nj&dOGdGa8fTJLs?keX`!sCU1`Phz!Y;wRV%V-SRK*yfE0}Gq6JVW zmPL`$E0ssW0daI*&71$5%a`bVReC*4*B)$WLtHe4prP;H&>I>s#tSHvRqAh65K2^F z#!4tNLxH1_ED|LKDyyMcJx}|QWw+CGfdxb8*^kb$?d2{si{rf3Xu-v4 zFNPFNA#A$UT;Vz2M%e{rNbQsz1!ZQ0o`)VhFK>GW*VV4&e!9}ISF#M0qq3!zV#K=yoR0IU@CR}bkx;Z z%HosvYS~t!(%$4#oyOFr&n08{O(xT5_*BztrkaXO&GHA4lo61uN3`f2HBUmnX5zmH zIS%eOFf^Fy>1!w`wLWqr(L@TB3eT1p2s1>{G=ytvZgvtI*9_s^?|vJ7W*A4tvUq0q zNrcG=Gv=BH!?;(Lj42jm9zrUUVKjEvyY48xBNGlIvS zI4s;(Y$R>r%_2kSk`pjh^cf^?K36Uv#55riU31n_QQTpo6R#v_ReEZOaV>MthKU2o z36&^wiiM)WJ}^HwEv&S`5N9fJxj&IeNS;ZyP!T7&ZnOt0mo1@t7NtnIh$APaapV*a zDjGs-!i~pO%RA7~-imW)GMJp17p7T{ia{%YE5wS&Z|gWZ%5!vsw}H@ulrNY*ql{3h z2rF8VYlqGng#5%}Xs?C-wpN6rOfO$RsalgXoHw81dQ)t`oe8>DayB)$0sJV@{hrU} zQOp%E)E^ z2?Lt{`87Ab>fDsTo5V^ZLjH&Vzi-G=$@)tE2pYNa+h9E2XNFiY?{tR9S@ z`{XXTg>e`GTNz^)ivwSH$WsGF&UE`WVP?&BIK~ZsI!?rCAZz*d&6}!uJ(Qxrha3fC z(W0WwqRLBhy^h)yoLyDOc;smv&(Rv3%~GJiec6E0gK~AS?(sBw%)lZW;PpG*pVND6 zei9c9mI!0A0ByT&rN|^wa1nj%jD*F_;(#EQ&7NSH6aeeq}T%d|eo|7>oJ!7ahf2+zmD~J%&W8 zoHCj^iqiFeueL2UTYWq(0WoQyG^neJ(KJC9T`M?}4jbIh2uCFXuQ9z!IU~%yVz_Ao zfApgt#0TE{Ryo$G@d94_)qfPlyLcjkA{hX0aMIBWJCn~K5zgRuKlKiwGXJ-}-v6<` z!o!ar6ayC!U_eb8bG)E<>jZ-VRXgQg0;R+?ir)El@t_nV#G+A3pM$D-EU7C~VTUYX zXy)vtQYvF|`hj9=2s0M)O3EiD)61} zkKy}2+JypVGF}H#!ZP<^# z>Gim5bO~w|&JvZO=5psaMfN{>ayK5`b3~+kZ@Ka^e0;}McwpZt{KNO}mk5%}l4QrE z-0A41yqC%R7u2(4z|tkhweRpz{QbQ@#mT9(Y~t3AB^Vl7D`mcgY+57>Od;em8bfP_ zGCC|e$4*2UKV=e3#oHmJuVhpSq!lYwoy~jYEoIa+DZSsb^r*}&F9Xr#OuiSkN0e?D zi(okBCAYA#*j6-cWv@i$`h8aT02+x}>4h9c_Gj8wVr{}e?A!||vn~?D;BD8RX7@^R z;$mt%6`w?U)O){Y|Jpp2V-SfJO&=snyTSmdL%R*6pK94|9%mMQs#x|5zWQbe$eWDE??VJQHL zFqD{$p;V9RhATwOVHL@lzaEf zzqFhp+$#R*n@=JTqKv>)b@&iNnwu87AXMrKJ;xkUu0-}~Fida)l65I##Wh&L%5!Zv zMjk6%mfi$AMaNL8bRvAE1;#k?Y>%i7-$4d)K9@&NYZC3LxDqb19hn;$NLpx>k>WsO zl~TCW8|%cTckG}n1lalXesqu{;uif-YgC$|B51_%B4>0p#XKuydrUHE8ULwH#2Z>X zacm4H%@m4wUem+B&xCjfg6n>V7Qt-EU9u&a2tGu2%%ns(OZQq4-Q2|j8R4lvmniw)qziZ*?G zG_56poT*uT9gjxdX+qTLEb&rEOjos$G+84Ws4n*v>W;2Pz?z0nV=tK;m&utpw5|Xg zjNCFjkuBHJ{KUGy&B1GZ7Qx~?*}h&BXt~*?JeEx!#$f&cV&rUWkuC8M07P$_$*^M~**v&9f4|3}9`@4%czznan#gU?^LJrrPJ+Zx1Tsa*>^q?Jq!g0f# zky?z^x22!LC<;B~m(YfYeitxoDMj(-aOQkk3}0(1tk_SvG0K$~OF_MDGSsmadOood-S-kbGrAsTey01GCkN}&b6wKa ziDI#ax!IgZGkZ*}Q|T<$4EABLvrP#vF>xOcpw`uUzW2m4`02wxLn;=*=+IKU`wctL z8js`FO{?+fvxn$jlrT9nO+IVib%PfPbvK)^7$0^vzlXpqdYb)s94oS)D~96@*Fqk*kX~D{uu}IU#}SOX*jvd?OI1fGeG{(C7$J=kg zo!@-{|9HnE2vjVz#DaL=2d>3yZ(WOCTFgZX_8dK7m)xr;Kz5I+6U58*tmtA+VAUVq~a z^m#R$oSMh-ktNu@?}z{#ZrHYwGS>?3dH5-0X6Es}H{U`|Cy71B&fvbso}=q7Ngv3N zK8oox$?{X;K}8QjJxSTFd%?Vn-?4BnZo*rpd+{`q-#~T!rn!%_($m?V7P(Qb;V>iY=_YerBHpZ5$Bs{&FdxsrIcwY4mE$a=);w z43(Gpa3(y%843Mulztw~O<{x_Nve2KpjeJ578)1LLK)3&j;VP}ioofl1}f92zWh8U zmu-aIyF|3Am?JDz%;2W;yU|&ip|z7oIOfXS1rK4`+*D^&V6~!9gu2`ti-x46yFl4Y z)lmV3SyLD z1?ZJo#bI%|P6E>ky0a%SlOKW^C}U0LWwg2lxwlvZvQo`{*XE+KmpeBh2>dB^zfpbx zu3SUz8p*ZmgN2W*`r9(YKP^M70C8Y$p z$`p;VhP&u#b3@6BNdub!UX}*gg~+*-N;aYl2%+2Ad%_|Y1OTTP%K!%8lRa@Y*t46vIqks2z$AyzC5UJy%AG*!gZ$Eba zGC>IsP!L|1PQ zp4mN*$9LV0YL%PyDMifb(Npj}t})E9h+qBXEAj3(Zjsju^&(x5h1XoS6k9L*Ab#^R zU&SL&@5P!G9k}u80V!pzT-Skf)d?~T8LVB>fn}{pY+SYk->6j3NB8Kn?|Ca$FJB^A zWWHF$C*N>GgPoSX$0ld}d)nLHg7LX56tli&1P+Gj8FN3yL|0cgx;wg5CPEph!(}3; zF&o3#$MW8G{KYT76W!#PSYxD4Yis#%AFibDFOQ%5kM%_uVOFHgMd!5Z>{%Qf8^d*5 zFUPyKT`pQa1-j-qWuQ;KbQ0xY5pP|$1UpuAOZFXOI3}&LHE+IFV5Gx6Dudk8yB;lJ z6aV~!osuaAE-b~7wH;_n`TsmSyYPx7Xv6S`F&8y5n}RvVXFzB)ew%o_bWG?hayn*% z=KjJuh9}4C8chSCZlm;KsB(>~4I68IJm$nP(RMjrta_y_IZ}eJF zo`&|4N&vX;r7K;?)8D2YL(0&riwhW?d=|0nQK3m;RGcL<8jjz4alLsqLxD(ia>67a zqOTICPa4giMtsc15wal*lsQLA3%Gn@7iBspgl37$iOfC6`-^*p!s-xkr1TYJJ}x06 zg%FrBdj(fXMD~|IL-Hig_cyqErtb z?#eJ8XGj%YauMqJXZPbx@A)-+@Z*1ky?b7ko{4Ip)+n;6N9;9UN@MHBLA>d8*Gv7A#f++SUl@tTgn+R`$cOgMQuJD+ zbIH@DD-`BYD$b)`XF?q5VofR%DV8grgiFXc1Xe#IbuNjs7|?|mGvf6MRVL%;T? zI8KJMtu=^W`RES1*Ob9fWM0?uyRjrW*>G5{9~Gq}S*G(T3_?-{D9l!hDH{maT{lnNiq7)8a zI&lic+(HwW*%gM}BxQ9hQuM^$!+7s+|1sY8ncv6D$Hs&mGh!$>*K8-|a`?`}&*NJUJ&8i8s3+C~_?LTrg3o^OpOK!= zqQ9d}p8F^M^sn*FdwweK*|dC6GA$w1a~owMz6rd-S2k(roSaPi&?5ySwk0ED+8)lN znKU#k19lN%QD$lr{x8$T2LkGKhEWHtkd)PAK1)bhgCdUQYG|qBdYX`nha4M7Pp^7l zXad8~lbt9vsGkfacVTM*DUoYHP_uG0h@p6{!V(>oI^wjaM@wfh zF!mziXAWWGldXm=gT79O z%JAJq6!jMs?wSSOjAlO4#S2m?2Ad7-Xg$Ovsk=BXMUf$TDY_=iLt9KA&*_+-zVZ7l zqfru2`5OJ7iBKR$2Cs(EmEG8J^EUb9pFQ?8&YnL*$BW{L$DYHrS8u|So+Ms-^Og9@ zH+L%i1!lrsm`5SIAX2rqMAIdd$T0G-lGzkB+PYNZ=|@!AsMi_;d|&_iJ-GKrd(qX_ zC3+jtXs`(hF5RHXoo<`AI)b+J6z5fEP*wQP< zI5D2am;d>3jL#%+^wbpY_~t|S-0yuDS6(rUWg~s$*s}g!;Ks7d05u8H3WS)RBIo)ahV-p~H2$R~FXt;A&`18C_`H1Jk1^zSNZ$+b|R zas1qQyz8@{L(QmR?VdfjeEm9%c6DIg$Oz_WO@;N|_|6N*@W8GY>Df5gwswWQw(ra= ze)8fuSfva-$7!^+r*Pk%BY5=qG{SU_@8#ze4hej`%!Wu-gwK-GY)**Ed5bOK*{cw>fCg9r3|yATEb;> z2hf>0M8>QL$1;5)$ObaZM7c)Jv84~GzJ64u&caTgmvSGMq8wfBlGBwYcM0nYV_0$W zGzA<5(fMHbEW>T@yU5T=x81sVh;2tw+(1Putwb$E~#Mpe!D*|5J zyV3oj=1#QnbMPI%$Y89$RvpJzyoJZmu4liPI;vc$2r$I;xH~Pw-!}?Muc6wGQls}6 zKzzw$L$k*-v8&sPLhlAlEnUY5x#p>58y-eOcwhW_n=Q@+p30rz*FD<#2K9-nbq@?L zMwAS~^h;buQ~CwHQDoZPY!ZBP6wQil>n4$<`YC*QA{$w&OXMZ|MwPN;MvhL`87^F= z*Mi}Q(j?%C@=yp9^V4|$_UqBrt|Vc%ZrO)NLctEe|)oEl5g>sWot`Pm8j}y&(S&591E@OZLkYz`gca0i4gzM zDZF%M0#}YM#j5^p$(HMyT86)#KRSjZM^B3wQ{FA()YLib-hUXST105>J$uQ~Rhga@ z)70Po+M96O_1iI&h@s>(1}*JhgieLmJ_~el(>|R>YexruwEH0JK$M(P7xI+-%%vBQ zN~MsXwKr`mAW2pF8MF?Wt(p}F_2GE_pl*1Ik&Z6WQRdvzO#wzsm!S33Kd*ZkAZ!R& z@!zfLSI+eC*=ZeRYqC$LVa#ozTxlL4R#WdDRZ&AHqls!o04EynWOxLRs$s8K`i%xk z49=w^W2cFq&zm&&2Jdu}-m+|=E~*w3LocJHY`O=h!tI#oR2o8-%mu0Q=ijpfNzi#s z3pENbKx=IC{9*K@UqZmm!wH1^nHWcv@fFWs%|tEGj`GMhl$Q=8+;SX|qtC!un3SAL z&)BO9i;lKUaw2it73X4W;13(l3O(2|?c0b+S)52CQ`XJTlbj0gXRgRNGGJ^XMVZ}T zcN^NyOv!ADV^~euMx%siScZlJJHMEt_(gwjC>j}t@taL06n@rh*I2^3N>0|+vowq* zR4{VOx5LuAaujI8Yv}vOp*ID22-S@t-MI!QD6^cRbsebIOFlN-gY;QVa4)4V5)gd6 znga={H@Kc7FfW*t*T@EG5}AuKl!h`IeCV*bNYliTSpZju)?Ne9@mW(D9Jo3Hb4#>mhT`M&V$sw+pa{mRR*XYUCz zPIX~!b={gW23$s{l{B2$p=_so(ci1yiGmjMoJ%7Rv84QD%MLNLwu_?RDphJSiCg8F z0bNG%XA5+!l3pm{o8S4V(A9^{h-iNEe8~Bk5*e|8xDAg!flGgrGt-!&@BXe{Y`AOy z$#{x>UqgDnh;wI;;A0=U0e|>gA0a1O!uvn+CwTHHmE9;*m5qG4j6z;T`7Ny}d4}2I z0==H6OlQCkJKV5=f$ly({pL~|!?n$N6-^?LQm!l&3-a%HBB8RkGCLhtG*^7kCjEQ& zv9sb#cuIHeP!EnA&-upq>BA>+$K4N-V`?RXU6$d;cqEBrBmy^A6xgp9b=M17y!MuB z@T<4IMxfIF`}+^$*#o=rj$5w7P1jt#sFD1NC@p&C6Q@q&pYDACZ@KvfZ0+quYbuUf zxk&fNMJgT^IRNEI6{?V*Aye=(!j9(_>P|zoP?ft?N!ilQgv(0|fCxBEL%cdTvvjoW z^&r*_b<2I>ue~RyH20P*vk|V>#ng~`DU%tEu-0@MXe`%fd2XJb4@kM$(yX(t;nCED z)I5rmL)1jRBQr4eE090BewD#$Mv5}<$>!#{>(5$f`EeCoxoQy6YmdU8UZ>2QJ}To# z$c0`Cd@i3oh(5|n40o2Cg)8H7e0?(Et!bbFWHhr@3Yq0sQsg{LjvQ$3T7e!qk;uz? zU}eq<6^H{-6r`$hYi2b=qjEpVeOXS&5_|zudu9W-d*&yrO@4)plrJ! z6a20KjC3DIm1Gp73Ut7&ovr)t$VPrHeUD~uhoehuF5d#$#qNkP5Abl3E? z=UojYZ^)0R^&6DeqsXAHkMQ>v7eV%e6~Y+1U8`DfxF>YYcl2mhhRsydU4Z zb2qwrmSS|}I+TkQfj)C-#L`YVLXLHEVjlPXbf>DpuW7>v-hD0ldj_z4=`wu$!|x_% za=R)^k>lkohQ-EwRxV3z#v{kZR8x3uA6|dUHq6bq= zBAw0R)Y%CPQ1r|7d0S@;rsfT60G#w-n&!dtssV13Os?zAYc%Z@KiC-^^pCT9Y~0uM zfi77o_F6}+Po8n9D@u4?dtx~3x}JTh2s1UW9W_5>>j7cacd>O1U6MkXnhuH7180QX zb5XEm)2B~r-uE7axFN~in63}G0VzFmJCSMS2FMR5L&~W*nnP4`$U+M|%FnjWP}=Cc zHqRbH|NK4#?KvS~l=&rJv`dqS@QjL?z}(1IoLssN=c*80l5x78lbtJ2UvVV@$wA5* zHHAwv_0wWZoVu5?FBDM7bI@KdhQS~#7RNvehf9jKqIU0y#n3^4M=+=v$9X=E@2gCj z+nShB4Wddf+0x%0kD=k>VrFi?DdFS7lpKWZ3zvDkdENjdpPxs@%OjRm+ACGtlpwas zgU4(~U3Cu~tr#iy&{BLChF2KSS(@o8GPQCrh4jEWOtrSiv*Y=kK&@I)PttH2bO`$7 zq|t42RuU8vMT>{gf|^yy^dM{2MQN7NL}qUIx)!c_0Gdm{^^gdz+hA|dKRbphRWRQT zZ#o`EY$(6Evf)^BzAgidWhU;B@(zI%_nqHH z%aW~F(z~1-X@+h;8Mez-Va=+Qc-8h%^mRqC`@nfJk{KzZ@em|4)1>BO5AkqAo8U;mi8s2M8#kK^NV+2?Bp4Vu31k-9BSA>DiO!S zkM6;3w{F7bHLLMQpScyUyJ;ml+ml$evPa(c!zZ3XiHy=hCWGBE_tHl1pvyf&*+qu$ zOwCc15c++sqFKBH%9w>qz}ERgST?(hjCLAzRyX9#jO+UxJX%_KikxNHO=4l`6_^~} zg0!JHqxe{44lqk^D|If#3R=6d<5fh8b2>xuswZlI(9p!GUcbR>AA35F!s_c~6Pz`1 z-_leiI@bS5Q9#+#*@mMO*sR)KF#$N}*%?)sbDBjU^ba=BY%5(?M|@4_ zJG+@;#cI%d86K@<5Nga5GP9GVHBn$xEX$ue+Rf0q3)t!p*?}{oh2k6Ol~+{VmI)qH@fO*r-q-wmS}5n)V6&X?-9LB;Ki&C+{O*SB zoAFzpd>8iZe-1ys|EGSe${GXbC+qm^pMDL8j~}6^KZp;#XQMb0j?zB&*I&b36m6#x zQOcw$(kK;LYg)+6xivLk#iu^=_qgjvJ5eEH)zcoq@_yw=Cg=0`%P;-_PwyH-XGfnz z9rH|o&JbYMd`YB`EhX_MfA%dr@yyT2IoGiLsuki$7IIm9{u_7Vf#(mPt*a9}%IpRnu|ihWF&zza|L|yj&EUVY8w5i1a4ft9DDb^fQNtf zs4U<@zO1!MfJgU_Y4lSN+c$2&bXJwf7V-tn*`?4bw$@2u*<#kAna?T&*?cCCSTuo0 zp4m^IaRiB=h1YLdgIb}CyMMA1g+fNvt34yj2As=$l#P7wz7~|e=U2audw1=_;L4527HSw9KZj#vKv>wzE91<} zbC%CNo>{ebD6&5>F^joUm98x(j;&aBF+Meeqo>D}e1T>79C`e;ef5w~&I_#l;7RRB zY=yA&rrZ5l5D)r$p~SA+_}TEtIu*r6O_x1toW(H34wou5O494O2m**vUjfPx!*L3l z;_}QqrsBqMvY{!7ya8rc41Saq5b4-SikXs@nit_jzn6aIq5fb<6ORS}O--jB5B+%g z{Anzk+D*~#tTe%0Lk+bmho^u8Tq>;F5#;(dA+u}?7Bp$3W$FeyuSZ(1$LXi%dX{5i z*;W|gPSG;=Xhn=Bh^^wAQx4IlG^R@DSup|<$v1bI*&@~?dotvAX4OHMvXpMJ9Weny zC~6$eNH_rIZ?;i&-D~xRKuy07jZ4lBIhR6jxeWqO*J)IA(8MAveR177U{0|TeJ+11 z^p+{K`9d}2n%)C!lt7&Zcd_S8`KE-O4BF4`MsjvSN|O#dlvP)JG;2Gxq!$4>7s&Q< zQ#qhvgQkc{g+*w}w8BbrSo9OD#i{NhDW66}q_F4>I&r0(s9}GOqjfE8X`(X?d6YI8 zPh0aFLbQMuvNOnJ7m!S}h@Qa0!ZfnkX>iN!EzHlNQp%uA z-yDq%FYCwVO)G?Sarf?nm`PJK(mH^)cKU8xg0_^2)vHsK0^7LnzFpu-(9)r0l7eP4 zv!W^grZ?`OXul1I4jsm$k3EUu;UTQuu$rzhis@NbkU-}rPob;b!R1>m99ze0uN)E#idYVjn3YI;E(UCUvc9VmQ2Qg8u;bbO@Ks-fGB8ngr z=4I1Zk_=$gz!GF>x4v}Z1Okx|db)a1C{~arqseHPa)I9$#qdBMGUXDEoIZniIEwz3 zPGo5}&eT*29jE91FQ0seh(Dd1%i!~0`yOrhDyE7y3K9tgls1LP0WklWQqE%)xg=#R zxtdLpZiv=I39URGP1nWi^zgZ}@$ z*JHzB(Qy)V{&}!&Mog%{<3>f+e}SGI8^&<8h#)ftkg=**F#)`oO;gcd_`nVfP`1Y? zu21~aJtDfMw21196D;n11OiQy=UN4|c`(tZJ~4gCSZJXpPkfFYgy{B&c2B5@I>VC# zR5M>(LbJsYRkUoZ;?DDC5ENC2A`uMHx%b5e5L$K<3azbrlt3$i(np+^NLi1$=~6xC z>g(svVeRQ%h!s!Epdk>{>>3aVzYHpJGvA4!+Pw;MtFFa#ERL)$m03-IBBp{ZFl`Eb z#^}UZY#MtSf#P{`Y;}zWbr$t@z2MlT``VI^p=H8FfTehy1(b|AaxMj-4`S-35Z(8X zq1du>lzAMEN{VLd8XDiVf)}n0N-$Ip1g$X-4s7F zu~j+H)%|DF)Ve;NGyvP}kY@GZIhio6-yqtI3|4fOuxX?py@^&j7ia?70JGWAx-vnr zAEqj_=G#+<_IJ}q+B%U?bTmPS*Ioc;W2-SWydC+z?p%N`!hxbWx6FnLn=e|g=RibT zLyJ*4u1baV?4#5Hn|mc@Q|(cFnnr>_r;3*591?HW)O51ZfXnes=G#)URqD&>v90BQ zIqC}v#kgAZys^{8qBQRVmKrGJw3DNzshJFRJ7 z3*fFF?xoF<#RM4zma#O+0mZedA{SU09mdWzm1@K6+ycJ;-S267h5%x1BSQBs>^qbR zoI10B1Bdn_lULB$f!=<(A7w7f(FVFhULae4d<9M*2$0R7#qE zhmRJp@1;X@pJGT*))b=y?W;$y+| zVc0&*mCh$m_i?sVkwk;D6@#8%mfo{_GOan+LU1E(I8(6kEAPAwPBx2&_8rHa`^ah1 zy0OXF@qAFBT%b&?CKfFmQiwtxw6u~fGG#wDzo4_{5W0ufpr?C?oOc>!M*JCVV45?_ za`i@MY7RO#TfG((3ZVcQwU7`TGQ^w*%c^D+c_$!yBPcRMT5gq$%9W!7;z;=G!M(=> zMdTra%qIGHUnbr#8LpxlB(tsY8Wn`LgLKRIOJZ3N@qeDdnOm8QyZo=F&A+$ zTh%=>x|Yr98LU3}G$O@w!kr}l2tYOkvUYwbj< zOvAzsiuuhJa_Bg76lQTo>;{^q!J8#RniXb^AeZ@dGg~0*h7!f{z)MmvQUb$_?n;G1 z=%f8RPQe!oW_iBYrgN`(&$q5r6GN1RhXss5Yc1fY*KPL|@xm7>KKPJsZxq9X-#d|t zjx3i5%1qJZ%nNgdz`k8ZV5;m5S~yM3(WpGc<_w1Inq~T<=&ICZ2~XOx!Z>1M&tc5k z#QcIGHK3xKKz_f&J1JL-%9xg>Hfg4ncbiyK8uG7;Sbv)IrF|lEtpw-z-qy=Z8Z4HU zX?i5|vfz0z$k#a51B$lBL-WCT0-O)P_vrVGc!tseDN8XcuNac5a$1pCOzVO0U>-R-aw?_loMdfEaY$VY zH{tmj-+ z+uO;p#e~wNUQLkGpnFi7!p!*qIy(nBp1}EYCo#7$gBCLOiFkyLSC;cnB>C8Ca-4TA zGZ6Cz;aD%Qw-#N4|Yr)c$y$D3ZQY%-87uV-y;swmv zgcDEb)L0F9aVF##Mw3XAMen23KN8i~rnE~ea&oL&5g=o3L=zH4JwQhD*2_l)z4YFj zuEf3*=g45x^jcI6Mbm7cPHY)ab;*&1)qty`bXvIf%Tbq<#p-q0U>0rM??3PqzDM`@ z#C#RW_FjQj3*3`3w_+iOh(wY>zwD?W@eskZP&A|()J9y&q>RR4VozSDqW&yX;B{YB zl>3YP{Jm=yp*K+Da#P2;`#xd2i0jD$MKKD$S~DUaRidD${&_4t?=~(g-c8T9j;=N zcD)gE-3W|-;J)$aNsP(LbvuZoi z4tQ3}Yfq>Gs;?63RYToiU0AS%Owv(V5eEsD=FqBh@U43!yza|DrU9=ROtCl>glz$9SmSuVYwOxqAvPvt-YjMo zar5XYokH;hQ&x$0H}#A}z}MrpbwuLH_j=J!z^n0jJtB=ocQ2q4b{!8O;mh1eRKY2; zX<2%r)oHL!x;m=Yskyh%J-w5Cy;Xo4`}gvGH8nW(SIIabI} z)|fyq)uEz9`Z*8^tJ)=_8EO@U#||j%b#9jPR5+ivD(9)sXJcP3D!K+C5_I6#Jh|M* zWw}ULt+=XFk$zK&3=Eg!ME~4Y$P}L7XJtjsn&cMO>l*2a*FTs1*w~+nPL^{NtQnk+_4%T{ckRt!vilK#T%|zgTc-g+_ZT$?%#dTA7E4@ zc$P*F*&2$f?wP9z@}%C)wQAS!k%6|Wrl5|d3$iX_lr3?|!LAiiE2mK@%+d8%;c?)( z8t&=(B5$g`%+Irz9Djl`Jnkb2^i?#mZLNXM2EXq24qc})DXzSL+nCtaT~yBWWF2rh z#8670H*Q#tp)OStx_{3REEJTWhs>5J8c4NAjfN+ue!|97^%~?_)!r;K3$W!171H4R1U8KCM|fcj6|165ORa%sl56L+LWVDw z9DqHz6|=E6EHL`sRt~4q)N4>pa|*0Vc?_L8h@PoK^!GVbWIl@-o9q6}oc~|LD-0X50Ck}2=nOECh+}~4YiqRy844nE#iI1jxbRoC zYvSBFh=|htiv&Yzefzx|r%6)AXfO(BPQ!~h4;N)CzV?6?u%cJ!*$npA$h~xy$D^4GKz~kWy6iDEFa;M=OH*s-VAoj{b%tL0V{h zKqf2{!{<+Adz31wcLJ?~Xf{!!Sboz;=_q(01V4NBn{k#Hdeh4|yfJO9Hs8S%MO#!e9ez>#?dph$46^EDnXQ!(tAx zz#Jye9T0J>AU)X5&R!(K388!752;j@ZaiOuWCxalb%ttW0L+*)LD_hMv-fW{P$lWy*Pwab;08Q{;ei6qL--TFNQ_$wi$%!=Is&LCk0hU7bsmOfsz(*#}za zNoF$Kny%;T&ysZ<@aIgFo`^zF*^aH7w6-n>2PCs?jYaXE8x);RzFft9FOJc@4oS8E z%{yf@OPp#LvMngbs(=z;hV}f3>MwB=?F8zQ<<+XRhMAh5YvG2gg{(}i_hS0P$g0{< zmuy_lmp_}Wr?5slQc7im$kZ&}Shaeu?ThLOj*TMv^Rafh zHx+tshW9$%m%iK-UM|hh-%!>;8K6>YY!FF$Km&ff#bO-!mro$?p5zv{9$Zxu_-K+{ zVjI*-#yn7%#w0n!s@X@jL1)>9dl4JDV>E=V`TQeX(FoH%?@kHI&-Jv~4KEYZSqvLn zlS^trR60y4Sg@WtBNjV7yC*Wnu3yz*#Mvxmk8{0Y1mY11f*OdfW*%l}`0#~;il)~c zfzk5fBl9NDVENB$R~oo|#OaxpR9t z=lq`MdEayHoiG6u-2d+X=Cc^w{A*kE*;@pE2B~hBflS&khFa#%Fv(tI=7&8-2KHBqleWt)Q5mz6}Qh z?v4}++Lo$K2tDPzz=3&j8%;E6FZqxpE7q%GOI#+|MT23KvQ?>I1;$A?Z!}5&76~Jt z9n*3hmQN((|1n0NEoymI8Ki$>9*W4t&`B&5t0KGY=$WTNei(?yW0F*rlUqePsZ_EN zQ@c9R*O$gfKaXsZ$Ylo6-Mt&lEuDzcpS?X7Col1hx2rz7MkRRaT39=a9utcvB*C*Z zglD75U5pWT(bPr9An8cQX1aCbPs1x6RLkqw@FmC%EeGT8GK?Dg**0TV&G+ZV;|EW#rRc= zThsdCF%|D}H~?}$WK%K*eoVxSKhWu(b3=P$hl~WrvdZ1l(k_y$hQK@xuYUFz=$>?r zCjn$=eL5il+O`h7hK!Oz)Dc^BO@8pSB}hh9XLga)_RF4utK=kIPeFrS5CeE6JV{ZNJ=3fS zco%X{^Yu46f{-_k5#Ou{YdA2{oX`KS3qK=+;A-4HvJ%qH0BeP?&dG zra;jh?*n+;Ic;5QXoMFh=YWons^3wBAfv8p@Cna!4fI4xNKT6&+8EG0HuoSVQtoS7 z>(tU;opQ2a`cQ9$s#S$+S*tL=S`Soj)da;?^+#^6zBhbm2YbNR%vo$Xj@||75t&qG zl8kE_S}V)a-oXLQg16LC-L2uIeVz$RUt0=M;E}nY#rxFFZtI)O)s?!fDK;*BNxRNK zrshUwXz2z~FXm2#0;8)FMv#&t3nubNClyO#cyPb8pO(N*^##KfDRoH&JFDjEqMblN zm$Gz*k^wKjl#vhwgI+2qOn;C9bu5$;9t%HKZ{g&1ggsBU=awW1bXR;ue8F?XGeypZBC%2c?Jskv;={%(f#sX zyc=RA0O>|3fl7D7Dxp;cR#1d)S7~*bAq8Zh&lh#W5){k9q9p7hd)$XAXzQX+lXT>@ zM76ih=S6jwdLpi2-2NR`pR{-cVz|COm#-denl+e!f?gzHao*{nmxi;7=Z)m@pI35!5u zG(jC`1V_*6z%g@_HrsY89G=?RF9^yIC6r(=Dl>QXZ(%k*@8o0@<*GtKt*so95$EF( zrpI&F>v+oDGWJl;ou(OAx^BgsbmrojA<4<`m-C+72geFQHdWP5C7?&Vh7X=m1*d6o zF{pjtCE;7L6XP9+Kxa)QZ7PE>Q=yvF^HsN6)iSoq-pL&=kT}6Yn!l5!J6&iCT+1+E zO;8nh4(X;-oD-Ng+Ayp-HoW04RD8&wCw*?H z1S2&s>)E*Q=iK~U?o{OW9NdCIJ(H{o(uy^-44LJIBxzWIvdjR3;(3xrI2Mr$n7b@~ zjc_O=7kP;~91jJ-?^&dz!Q(6L8WNiN$DG-uwr2422~8%GD6>>YrhpVBEk@7Dh^wsjLkw$8 zkBwk*=M==EZIqPQSG*0Gkr6aEx5@D?Qg_9UlA;1Y9Z)nHmt?~J8N4JHUO*s%1b0T1 z%=YbCFGR~p?d{YBQAHUw(MpNLid7|%h}Vp1XxVC2NO)fEHI{RYB;|Qt)K}!Dk192k zT!(3DZiFNAZL@AH>}sMFSE;U{94~+HyH)t^2i_&TlaHOV96S1l&@-6!7p3*KIj(R1 zW7ioqxxUOg7dJeHVyJjD5|__XrJ{n>pt&RSI)r*1Czg5S<&A;dt)}^At~quBRJUZ57cywPaQA z$lXRT5)f(gVlFE#hPF{#c)JdtGnRBU5oCZ!FrxOC9ubeP51wI|%8bo}0&u}2$zMNK z)&O#(0Hc?_9ga>xxbKc4@qK zE|)VRl~S(c#oDx7E(oifm&6>XIpFZ`D^-uw z$BCpYf{hjMD9^T)ISv24I5BedPT!NiM_6dKHYJHMNov7RR6J35%vE6daCU^Fpv=LC zztiaG0KI?4N4R;3w?vh#Da%L|w8bLTFlEHW%R4?+LbjyinNAPMoSY$nB7;JSkoOi| zRl&)QZw~$}{F;2+!%_Zk2E!wLnA9;Djmb%9oiqdCRI9)~LqX=S4AEoqiZI9gg>Z&C~>yJrn{?R*iLN;euhsk(}D1@6e4-QSNHO6A$r zRj;QK=lZq;^|l4B535vT6BTR%hU#oZOG&+ur!8KyG0<#d+x4WhZS)SOaov4S_(aCf z|J50k)Pq{0r=Do0wavx+LR~8&`{StY#Aks+*%DY3NSPm|U7L@>_B_=dnL>s-CI@y}KsL~Ru4(g3ux!*TKTMt$%qT--9(cuj&nIkD17I*Xsa1*!Jr+ z1+Qz0>%RKWhEZo<_1Z`4k9#v7(GCsu$VFy)o+E~(s|&+7(2)_2pg}D8JulK3m(K_w zMuJ%$S;n;VXvh_m16Ld}OE*>9@7RWh6MATr18soS0SssY55pw6m1n~%uc~6$tON=@P?9-=dp$aLY_HuHR%RrOu$_ax7~~o zi3P(l?qcCc5$j{Nt*CD1smdQ!EhPgM7!ARm5Y+9In41xA5IzU*M(lD~x+Ry#X7ruI z0i_b0&y0yY3*rf|Hd=NpjmcdtvbSR2kqm2Y*?DwOqHAht7IIKFH-=mxL$BeYpUH6% zN>bqzeeVY9yoO~r60U}&vx&yzqPG1JC5Sa8;{qz!xpgyo`?jM*ug80y$&=bq zEiv3NhtAn8h&0D!)Me;Bnd>bxIaSknWmt2G0;$MG(0O8_q%33Qg0l7QL@fhN%hd9m zy5+a3G78M*b7C^Y2s}}vWr%V-BHS2X*G=KXbp;JPyCcl4#}9t2Va8n-b6y07ZN@3NrX+`P;gqA#!y`k!CM{Q^x4f~ohQphp zZd*kn*?_jD7Nv7kq0To?oora~Llm~a8^x+x)0a3gYN$t7Bf>;jF<>fGc|Z{auR47` zIvbUEna*-rm4_Ce#Z^M+%!t7a_|TbutGehlOF zlgS&CzONm@z(~(lr4(#gzKf#aVXLLu;EOCO{trB}Wi{LoRr=fku*?V}M!dxqFV*iT+DsqPsHz~o*aXD{OS%eXn1`9DCWOR*jJp_#73CcW zuR2)bgnn^V#l97}oH8g1Q9xy3N=4XGoeICcEE{bl8zGgCz>V2mP%Z5F_4)qq8|Xta zSy55}^ck|05ZYQhVHY!~6ozHg!*g2R@o-h?DF;NNG5OwXpKCj6EBt~YFKq!^xb*~hGnccf6i^_8J{_S1-^afL#k8Kg@~>FGSm)^=ld0$cgnHY+c%0AHg6S8HC9LvfQPI0Q{_^I z>XNeT0z9?q0|SW^AoM<0o^u*bIrdnLq_c487(e>LRy_Rli#|e280m&8#sq;Rz!Amc za(rgE1}tc0ZST6fYQ=oa?^06zoA#%1+v=SYJLq{n->yP4fjSzxWdv-(=sg29Z9NZY z!kw;VX&Dv6^~Fh*n}ne!18e}7koPL&70S?eOHqQdYJ_Mg5kOO`{Tj9^7+o=?GRKX% zEm0Z|C_g0WmJIxF{Bi#`eHO>5JKz61kJm&$fpV@#0-}%FaBDY{q1o$Pfg2X%sbdnn zM7=cGjJ=$Gkf}TvNsd_%fu3JW- zoI|1#lUX~XpqO2r%VuFwRWIl-yCPhWCaMBCfjfBAk8pDbnR7 zQ_UD0$l}NUbswHxyAewc?ZQW|dN*#k;TAmFKPcZNlVw=0BM7shp1suQq>xB73iqL^ zCOv4TN}4ht9((Y19JXXWZu-p)_?u7tBX(@ti!n+nc}lWNmmh`ZS<{iu<<#!UsNX#( zlu$N3B05=reaVF=w#kl{u8Mx`*m4c$<0GZ8~15anbe{F%N>8f zmoGUR^Sat`>>)Go$#bvGMP;fkHT zzqvCsc`00>&y_|dJtSi{e!rNIQ)_D2hO0=N?a35YELn&x8#m&~7dK$W)JeGLloK&D zIDp62zAU@AWFjer1YZLHXF!<0;Y8FYl?6B#(Q#qKpQFb_0yYt4^8h}6#$h;ox)Pz> zJ($6d9@yY>lX!l9u$ba7Qp;8QLcTWzj5YABsVZx@9g)v>P#n9H~@zCz7IWCn~5V)|=vehnQqhHN=95 zfnZEbI!tV z@A^+X_4HcYzw#a|TrdYW{rKznhp+xCzVVNrR5vjb0INBTa!J!`M$p*NMggoq-BeB) zYtet>)GcMhLG<_R7KS{JikL~C9~)I%2!WGU@aa!pfvZ1s5x()S|Bi2THjj2Ge6`H*K}!s`4^9Cb~J;wT`079wZwM!IZ9Mx!DH?`_a_VfwSL!E&?5`=pPyqX21Ae7>=*g##sF#eUF*c zwfy}R7vkyXU&Ie7u@)#%pS5%eKKa2*v8{Iy4?Xz|B4XauMFBN|9ecYmmd%JFv8xe6 zd?_#=W;9pA*M9N~Z0qioqJ#^DL@Ff-qcxep3CoX`=RWbw^B5b+V%n4rk=ftVKOzw3 zpFR3KK7RIz=%ViC#09hP<7+O*8mi=o#H5q*ioL!dbdxG1k1i9)Eaj6(aGdlHk^8Ax3)5K4cDZ zR9IdAhZpF525`u({nVu$LD!8XAdK@*I{}BzUm(fv_WK{h#$9`H*@fp})0VAR_v{P! z=oRnAw*CEh^yxK7#^W;1;T_Pm?>H8Vw1Z?{x`6NByB>Kez8E;+`Fu}^s_`_NB#c{B zgx>sN2X4R}f8fA!J2m(Vk8Z$c}+4MXub31_lLMEBp(m z=DMgHJq=&^lCkeqG2(GG|Dj01Y*cPKWq4p{m0%A&V~F5urQ*u`o)uyFqUH*XI}d(s z4#e!=ZVy(pWVRvWBvWJWZAY(7j$VRk(>id=FK(w!A&D=2?i%dx z>%sTF^$lc)dJv@U!J%%cl*!@nqmGb7@(*AAKECnguT!ZyRnEcgT|@ZhH@}E!lN&LX z8Nk3m8fTw%0xrAaOeB*r{QH0W5_kXV9?@^V=F?Z&;{n`y+k>c-Fg?g2S*T! zgk-M)fv@rmp3ZYDj3~WEBZCg;`p8gMw16u8t1o>!GDH0cB|5Nd{&alhhyR6>4xNqZ z)9Lp=`4w6t7A`sCc-;EvleqMxW6;^sjG67tl2}ezG!uXO{r{riY2)*kUw}P(doW{0 z7Z%N$g1eqshXtK&_#Bb`jQW*^B2(T^UxB3yXXBQ;S7OD{3vl^6PQw@e^=C-X zIXHdEEX-|=W67)z-22#50^{Zx?c|meC6W^`xj89svuj`s*ZqN=BkgfSG1qktI-2nu zP6fhGiH^k?tJKYV3c{W~n35^7YqN60)sCp1thrHVrbUT~3dU-rilU&lDm7Vi*YFEJ z$5n_tL30VGOQXnGX5Cb}Sqk-M3H(^bfR-_VZ^&PM>LJPFK(YUa4v~2q6@-v9yZ#7H}7kEyx17xuIEnII1Fxk(cGVT*Op? zCQ=Ow*~K$@5wr~Y#$1MY+ncgvr zB$AU?tiW^6KacayJsXEDUWEJa`z=25iK{U@(1&Y3@m@UeyZ^@W6OO>yXPk(8@A?_O z@b@1_GUecNpZXxa^X;$D_ZSpiAJ*D}lgIh*JPZ4WGWgMTH=<$EJTx{pBbIES;~K)> z{w;MXtJYI8jZjz7h|^C$5t}w{$E1z`{`t$FkdEYv3(vs!|M^pRc;(}`|Mw5zOP~J; zF1hd&?C;r(>#zS8N%%khk6+-B`SZ}y)+A^Bh97+sZBtwD58wC-_U+h$Q;%6Hof6yR zI=;QKA)RNC678&xc5L6%i?Nc6cruA73p!Ggd~E$Icy7Z+oN?TdIQh84aNTWp;j`bn z9>MY`uD;+b=}7xWhUxs%d7(gEpbEb=Y~l;=Js;os=%x71H6O(JXPqIOoJ&q$PLCbJ zC%=0gzW#6jfjN`gvEqcI!EQ=%OG%_C#a%WSvvJqMkNZT(FeUO9N`mh@?RcDZ^fC;k z^Z3jUZomz9+$q0j{L}JfhawP3;5W}~#qXcnh=#C%6-UmOj%a7^0IvPo*QhIb06=Jo zRkXB?>+yP|K&!c^iUfS^$E zfs@URQe~b%A`z4KE--3^_Z@agF7Z^H{*3~2xm;F!5A7q*K_VPaN!u=@G#tvOe9lmvO?5{$RtD_{LKrcx#R{cnB*quDDlGCYPm?)V)(`{~cgXIpmEGTdCGNVI_4CtMIiieFoS5-9O;! zkAD=QXdB9fJe-75f1NR{3p=;(!q{LxnmdwMbm$y>@|q7(*E30Y06+iLqm+=lB4>qPu$w9hWdZe$79j zd+Upog!A~$cmD~8o_G}H%otiaTIu=AuyWOE>>tTU2ghhR52EO5-Ush(1CFCT>jNU^ z640(}7LQ^mNgsa53_P>;Wjy@yHX&c#|L9}5;GB2i1gh4LT)G%7l+Yi0;T4Sb>=A&( z(`#Qu&)_JM@dj+6MB35TfazV6@Z-DhLm||F-D73E^72cVKVvc-mjMATSQf{2!!al z`-jsM-1lR8a})iy7bUvb7zxMlc4j!U0-m8snGMNCESxzF_dop-uDjzt0iG{kV-ZP;DJZ9>0Cyn$7ggjVZ-*_cya3vcJ`eybj?>c3YV4cgFL7KR=> zREQNTd7s2=YN>WhL^Rd@Q@J7rv|@%MgiUWZBl$;L)w(eLk~t2NO#aWcB5y@9QQG-I z6tn$wp_UP8ScJ|Fb}cMpV5A!y^TR<)tX{L3ej(qnN^c|(m1?G{neE<%=7Am^%~=Rq zo+q-hUuE${_Av>EgNh|j7i_8Ih!zzuaCuS7;)Se&%ZQ^yL?M~P3!Y)ag)(?A%X}DN z>>G-itDvW9i713f3Q(~`1JZqcc>m=e!;L?^2g@ip9e?a$_`6SDfrSgFzql=N8!LWE3m4wi#m+yv>lu?U*XyUFir-Rmd^JS7e4|;x#VY~(q z_b)&H*MI&b5OX5gN|ilVAw|TSr(){V1%i&+y6tiFY`!1)k=;}czKC7B)+3pSqOqkF zkysKT*6i`52wcNAW0ZlH>-d^Y6|GeyD-zkEl#pYocpi1}GEo2|wxD2a;R)nrEWxrT z!jNZzl%a@P#cT;d9ULnB!{|H5W`H<*noMn=OY-6QvzI$77CmZ`s!ya zKLvMD_wcqW{u;md)t$KXwBs#8?7K=wIF=nu3$8MapVmUh7n{fQ03$bASe5~8B1%1P#m^*tWj#xMcD~?-= zMbo>4u^yz)7!QYKRI~EA7qNF>2z~TAa;)h)eC6^BaNAe^4ohZE5$&y2&%G?{@)M6f z3d@gNhIgNR62z2&uE8i1OBoWI9mV_KaUR}t_Bkl$aze6UF3hC14zwh~IRB(&_@@uN z1K+#yLR?54-fvf}#meVj!5F>or!G1NU;OZe2vIk9%RP_d(zl(8NewC7ONn}NV;pZ^ zc8H%yw(K9l*X~|})!T;^LXzH})f+@=#86OjNdV3poOOAUBdoh6I&_AwQs7H%X!x|p zX0O4&jT*1UE9?a6WeJyzccPXiGkTg^zKyna2vjM;(M6f!`FJ23FR=*m(p8T${*q7r zfAoQ7F*PDDP#9mxjdhPTmU3iO;3vSYoXlVt`v z=+Df1ND98csxK~Hgj=Xd*VlB1ZDEpg*P(6>HIgcovJ|sQdVIzF6;noGrD8@1gTfl8 zpvG=zZ06!6a25Q}M8J zvf~d;pTD`84X#@7z{=J5^fe#Gz~GpS7BBg$cj4cz|2;Nrsv=!1;RiqX44!&s4KDtx zQxWA(ZzPW+=gh_hXPtsOA9x5$C_z5EV=s2?+lx!iK9dqoJFdU=4p9yWaWDx+v0-l? zesRa|aOH*P;iP%gg%%1}FmDF#TJ|DM@wM1E_5b9~2k_x{pNDT>^*&(^-u2K!c4gDdM$IEmGp_hIM27?2sn-&}H@IM_Au9<5r%Rp*|D z+aG!c_doI^C8a~@8n*D0U)_$EH*Q2TU0=J?Svr=~b!DnZQnk|F(x^H|x`tx0lyo&W z-@lr=D@#yoL-hH7y>b`U?dpXQYsP=y^B9g^vXE*B6Tf(11IDQPYNq7VWL5FWi_i1# ztaQGF+n(NnwYx@?4jK0#o**Xg7!0Mt>$)|IUE>W$UQZs^(siO)8;n3?QcbTL8Lsq;GVgj{Ry%p*Ht%x?=rS9axmH&;GUfzzzrX;3KYsE)C@(FBNzYagY^_RHxl8eyV)+&%p;a1S+Po+}m z?H|=L8xX92KUK@wY+n96I+nriz1_m<_m-Gy+1Y?EeBoP^xT?79@=I{RMW;*GG%z%T ztFHbcetz4NIP0X7@Z8!R^x8lJC7RK62@gK98L{v*+;;nFq#BdB`~w#VoByU;9>l|| zc2IJhhp&D67X0;9=ium*mf+q;p1@dM*~k9q=fA+a&NvyzEj|?K!8C5Y<38$YN(j+8 zXyoJRDnAVo$>arhV=$6YDoo=aNJZb!p$a`IlI6#1a{e ztl6{`d#Ni7CmIAS!H{ka)_eMTaKkOPVzB7Qd9^*x0u=;%b6I@*=G$=CjOiGn-@nqc zAK`{p6x=dy`1SA6)!vSw(P2eOr2yYu8pg&w`$U?b{WO04$Ws*5l#0U#zw;AZvf@~r zv2+o29YM6E;`rG4rwK{(`R)7h_)8n`2z7m_=4L#o9U60)niJ)8`=dK|6>97jlA=p z_ubuBovF7biHH(fRU!Bc%lxy$jE#81%rSb8q<0)0Z~1-T10A(+4be>>jyGY-lzA8) z8pfXO>*yaVh$Ol&Y0@;ow)YP7OGnV$+J&~3L+J14AR2EF%`M&s*dn9hDa|^f+rt%V zF|Ubzl!)v~MJn;(kzNWQ`%uc~(U5GR%Fx8n&^{$cK^JOvYzR#)b11RRLz#k0f8P#4 z`Y6(3P6&a?=E-zphmg(mpg_^Mp=k=@3}X&cm0lR4;8aB`Jtj$2?C5Y0cJJAQ=~LQp z;RWZ=_v}Khkj0(1--V|heG-Yr7Gy`Z<85c3hNF%;4lD2f173dl6`XMDnRxNV%}6Dc z4%ooJFxp$$)Tn^HyS55^xv9BT0@U#E2!g=`f?|)$H*OH&XiNx~9g|uyXYM=-z|4qG zV|RDAj9OBW1g6bgfcDM>BK>dD>+&8coz4reAR05!L0t{I@$K&)rs_EiCrI5^n}tLh zlN(+6?jqS4k4dM-QXtF;5@9{2P*b8l?Z~BYC@??z>{`U*2@(BNlIrTBFBgi6d*CQ@ zCf;L-HLP|sEENlaDq>sUxX9aEYCOW^Jw9h_a>mY+(YSI!;E|)FWZR)x^UPWoJ+MHW zf{B3-D8>Hu>%4DEntdYlhvHQIVqm1k+KRg^uo z=9(j*StsPDx&T)0E>-f#We4c{D_%1${j^x%OSteN9E$iMZeH|3E@9CYb!Y zfA)7a5utlyZ?J&mf(W7wJid3{#Fx-KASUgGmE4hQJ9Pz#`{zh38Ka?X@ff{$(qp9S z;lr!iT{fXgN^l6GY+K@+K16|@^`d0=U`5f<)`?t2kpvUWA!wXL9m!1U4w}Vk60$Tc zNl@_Hcx`3MB74PrR>o%>sCcBsok<~=rs{r=jJcA@HpGG!1^hh}gq4Uj!-HGsJvzE( zQDE)C{+?}gLq~)x$Hc)>v4V!SX~Jqw5ACLaokvsabflUli-4S%$hK9%$l)S zd`kl2Veg_S3-f>w=SD{RF-YfxM^J%C7?b99pk;c4lB5?6GVfAyYF;v|v?P8rhLGDD z=s4xdRSL`=pGOOAYFP~1(4G&Tz@x3$>N36%$(@iOl6Wz$X1f4n;j()*T0?%R@%)6fiX>9Uy| z7)sy_N|Yxo=)#=#EVqBXMUFoUALLBei4LK@=F1kG=xGi0a37^Cri^f z+CHKpU>C>^i0vMNR2=Qv##XT=8e1l*a|!pLHNWYJoz{srI@;017?*O~1C2O*sIz;p zj)Qql+;Lkqb>~{TG7p&4Oq9kv!O)36|AE4e?(6nFZ%s1sNRHKFS%6V((>ih55zDY@ z(-!n>+(hA{K|{e0pnJSEc_kAFiO=5LI(kY41m}1+f-QHFQX2 z$TL+YF*ec#8%wlMuxmhqI|-Kdu=PAu+o-~W0M~iIyV`p~WKP7V+EuFt-4io3l(mr@ zdXaKP`yCi+R260aab|$C7YQY^U_4k;Dh?=l>tH~6swg8Oo>T{9hm$V^M4PlkK{3fl zD)wcj8>S)ELe*UnEzONGDjN+9ij)g0Dc4MG3<(enF5vIZG9wl%28W{(toYArhHXe7 z6$w7(06?+tg>5cAEUNnB3n&y;v+kBAmN*}xE@ludZOurfDDh?T)Oqa^DVKP>MdV6? zRzyLq>3j1yf|vCppwHcvK+~j{$kDjke9B+Ox`tkeLPl633(sR0`u-#NiUcMmqRD48 z721utCcMA0^vs+)9S+pxsy5saa6ni*(#1Q*GPV7Hq1^=yk*!e%K>-Z|R~@OcKQ`oh z8HVaO9k&)BT|bFv%_9A83OrljhZs7 zVH_}KdiCx?xm3Pdw%qYj$pH?oY^m#?Z?}GcjXx@hj9bH)%Gw(hU5*In%TX?obZv)1 z3MI*?uAnF=DLpSMqQ)KjclRURpFu-Y+kvUz)o)49FEg2%Rh%75jl&I3+){xBL?c|Q zGW+h|2Ww*t_03IJIf2Ucr*-EoPqCjSkm>8qz2&_DYt6ChMr#vJS+opS zPn%5930N0YF1giWQ5Ck)u-;p}wVxmmq&D@y{OKIzn&Q6?M?=CZVW&QWs>lchPa)sY z;mPLN2fYDSFok5Q4atUP6wB&j3-SoV^s3=%-4f5$%RyKF9ao$cA;p=7b_L_Ly-}6U zhQ~sDf)eq7oLC-_u;e{UKQQ#y(q8ubo_qsv1;U?=*OUPt-N06pu(oD~%&srKoE4a*V3%nkiPj0U6(kmRD6<9`o^W z)$_&o#m3N0LUL#8G-1U7<{AxcOkRZ#23h%DA?%IdD9vcxm+UXtbm^-Bl zvpYM`)zXZP#wJS2air*+M_7NZZtsz!ATpX~Nt+ycN3+;HFp6FMBiJ`I>fbyD>dm-3 z&mZmP=eXK8{nhuGdWu;?N_n=tbz>jHt(h6Y(@yizGN$SVbfI;i%cCdr@4m(k=Cz>5 zo??WnEhLn0y-FzFJ*7GVQ_IjW$ux-meS^sKrV$mpShIdIH=z|Q10M`0gAcyz7%ZII zMMYG}ue@zf4!-JAvFC^qxhG2P9Z#O<#KHy|6TXwWww6^$2aAfmJMP<#Z(cmZItcEGO@KLU;m?6N#{CJ*^Osp z43Vtn3%FQz#ZGooLxQq#l7ep|JuV@fiXwLkEL(4>F&EELRV7+$a$VgTtGO?SHD%ZZ zH>}=Q2#!1tuLuVr7%&8yoTr<*RH{hO32NVsAbma#(!9VIa%D)_mU4w-iN}bm2$F`f zh~>Gy(kfCa$Ba4)MR?pbs!X9+vOLHdWKkK}aH6xsm%Uasit4ZcXcT3d&8K+D?>Gf@ z8=A_8*{Q;Jdy|uISB3Ofk*>m|;}%8#zyNhcjZ)1pG&9wZ!q~_#-8eb={1kL4!2}|4 zF?R}8SwVRWBk8>u8%<+qun}{Qn2Sg|B}6Fcd@qoABr@eaug)iT9E>*O<0Jmhw(_Kb z&u9+#-W;wr_fpAHof8>D=>3{yjt_duFv=0TVX+L*q*O^kuHp(gtlrGY#H}i$HBJh) zvRw6SVm(()MTDz~u$x{&^adcVq2&K&R z&FqH*R)s!iAk>T*GnNS0gH;%0O~L=Om9dw z#l~z5|4;XhVszgq;zpRSgt`vnHFhO*9mJdxj+~XG1W(0oXMDoP<_hc$q>4AQ?ql&~ z#$l?i4tUki>gEW=CI>0Ge_0tki_6ZQkMqx4EMwPvA&76^a4#-@$0BUlUBFL&vqsGB zQXvPQ`tYeZbZ#p`bbQxtJp^C;!S8YA(X+5{(INPoZ`_S}bCS6FuTRB~e)=fRJ$5FJ zKYBVPBL^d+4u1Hvr?9zq7*Q(9-%71Zo_sLj_zQ>5QHjK@T{D$R7DrLXbZJ);9&0@w zkL>S9?~dKb(0?r|K9&0OoBuEYlah&x@uo7S;nXD%6@a9I)>SS4Hzdw&uF}McHzL_c zRk0bRN@MvQ$&TM`74;NM1Uh zt7ygKumoAI{>2$EprrK;D=Jdy1_y#5U8&r4cr7v_NTk78z18)7$n*`bx*wid zd!ya}-E<{Nx)n-_>@gptw%H1r;`+y?T}_jwA{q`;hmuydvD7{f_U)lACMw2Pwjkez zQBLnhWX=+#rYGp8wI$G5hVr3?W>kB2xi*w1Jd@i@;SVU|G5L2}jf&*)LQYlRQMfr( zh1PK$OPPGx%0&;}4JpT=tQPA=J-pO&<#T=Sc0;>uD#>;|X5hqBh2K$_b?72TCuI}k zp{=s(GgT$-4b`B<5BXZKWhu|cG$#M$_w@X@YM$bT@6pYnIDF=?FcjI^MuKbmDtmp|;$*6~32L`5NW@6))kv zfGPUw7j3VNM<(lT* zk*WD8t~)Uoo=51mu=axjq*0%vqSY71Ctu#Hj%93lKQ1og5$#Ry>y8~fS?3)!GOy5c z3)wtQn482EZ(o9&?^}n}&%T8D3l`ITS;mxR3%mQcE2RQDU%)k29)YEE+Hvi-?tob> z;JcrF53YRsp|I>>w8hlD5H<3clA=O2kjIQx15d5qjoa>f1Yf=SEPU+JV{q*c9+Z7x z(67K8J&Hq}!{Hl;^~$wN#cO+WBc`QNzP`4G7VZ;eT}$gmc}IZ30)nD)6d}*@l8)=8qFMra?XWCcLtlinR5B5Q5*T$2>vaRi zL6ePPj#mKI109BE^9Yx19Mp(Vbw#4&ZAH|+!7n2Z-X+vWg#&v}-G6FZCw1O9nzR2< zVX%PSZM}#gA|qhic50lNH)OhG!gF!wz2wH-y?E)BU4jtf?l~MVaM*%*XlZJ|P$rK@ zp4o(fp;0t6Ds|*YfPIxzVOK3%XpR{;>DWaGgu@!2rlOo^uhDT?GQXX=kt%-nn^%x6 zxLCDe4W_r!H5iH@Qyj%`wunT+z@c+Gv2xXR-1SrrDR&5Ko?AnIHw!yrY5u6;ZmCL;7 zQk@HNj~u^JDKEz=l}MtcWs+F^7TL&& z5=(B(MDKnJMM~sz51oh3L#K!$D<@Ykk+|LxTkKm6)e#sqB#=V>s1D7;%6LJ^cC}!i zEbARX+wlz|PqXW0^?qV-d>PkoJk9u_R z;M9?O{UC}uQ!OpQ?z(Q>$hfvb=Y0{|sqj%`vpLx{8P=c7if|{u3h}+gFn+tLMzj9n z-Lsty0rj$_k5u)y|kfH8*`+gXC>W&jZ-B-Rg?6>~C z{8Uw!L0{Z5hVT7+9m0v2G8C)iar~lgeCXoS(LO1GSW_$Rx%U+bnzO`vP zf-mpD)RrVZ*4%`dk%%ZzxQYnrBKbE@()^CCk5sPA@8=*wOl@wd@@*Y6~ znyvT^?2F;Ju5X#kqF9VT;;LI@R}57J6$?edSsSKi@(8mi*n(!tKC>;Ol*4DrzM{mrhjk5h^=Pct!&o zEZub(K31O-6rT@;6y)=43Ri^3hT=2MoE4_9iLJ9PI9%w8-xc3+SGyicJmx)zRxy@f z=;*G*t?>ZdI#i3_5bA`l6l;|eA}13*-U{4dc)U9O3bvMK*M9CC==G@1Yufx`%Yk0@ zd(wy=)Lr|lYtkR;l8<))#c#f>ufDJ;&}?snJfR*H44|u3`axJ zU61(knhP7{bB{VysToWml%jhe7E+)ZdhY5?J+Nx#?jD?X)KNHg zUOTpp7IDkX_fVX(@vom*CeK~7u@|>Kuo*_BA15EZ0YClxa`{@nd|(^4?-@omm&RGk zXW+=gPeFPphwFc_mSTc~SSawCJexmmU*gCr;fR(-`SbFYCfpn{vE!B9_~6-RV^w*N zs99u!W$anK2ItH@9g#!?9rW0ahDHoKyR-+5`zAXR!}7sBrcNM+7Z?P2q5arB6tgFX zAQyxs?GcZUOpVlFvBzSusK*N4{ykUYctZy3P_^h3%jBj(!6l?B(vG*C!R>4 zM3r_VoTQ4KzVv9XI{BfvbQmVXa(Nle5Z~OGbRpb1FtR5`6-IJPUqKS$3?0>UK;Xxn7H)HTglVcuQO?$=fR z8nQfhHPgU#YDP~7j)A;A6&DYt+9w!By;p~!?UY^p_omOeL0w8R1fgb0SE|-MY8PSX zl39YI0pA|C{ z?N{xhKi2YvCf{JZGTZ~XA&|a5XeqFY=>`#M?85U~dSTwQ5+8g2={Rd~1DZZ|BCfyr zY3v=yBOZ+?5Dvp@Ol#Z*#hb_Q@#AaKj!egY{(2{#Soc%Rn>!WHzSxg+&c;pmZ^f@y zbpxfac)RQ#is4JwKaY9ywhJ9#?Zz<#;!(O5TJZTFK7qpy+m4<-7rXi@h$WNw-c8S; zrEM=2#8qtHJ%)TWfrjR&4|;Onl4MdvQ>qEm8=K_MeSCeF9bA3hIr#NUFQLDuU#3FG zEISf`qmIVTzCN_go+3!Ewx&jGp;#*(Q`Vd8Obm^3Q8Q|9G@|FQ`erlh%;!DHYSq1% z!l~engvU*M0atil9|#Dfl3&}BQIF$KIg~k-;+&MZI%DNmZS5+sT_$%?u;tcMxEXZO zG>Bzn3NXs3g-U3coCNm`FBJGGbVAPR)hsW(kIZN^Q7Yf`?Ws=2?`oT^Pt*6>_kU z5u3w1`moYRGF2jVt7?zK(h+RVB`{el2wM^+3kybN)CLOTrmjS#+hT-NQ0-hO^#}{x zuH=x<7FDpdTomdsx9~b=ShsZ?5JYm0WBMGxcJf>GK@i zaVRW*H{d4ef^3(qau+J$F#?uShY<@|GwMn~5qyzS)0 zxbe8@K)bMf~))TXFUKPeQij z;?DawQryYm?=C+Aqd6B(-@XbTx#(mpUN{|ly9aUk#qD_Fxvf$_yXHdd-#>y=Po5*n z)a7CZ$DDS8_$ei*Q@HG};%IJ&W6dl5_{Hzm%5+ACBd#*#`%~wfF2#;%O_KytIh-ED zjotm&zqcDNQ5r3?vIUzM27&&OQ4G`P92p(My_D!?Oln1Iq5(mQGv#bXIB;)1nONSk zOWnLa3I!OSNPOgst11UWLqsW<#iDTt_=2rjS@~2G)Ob8#YQx5;& zS+N}au0@UQP6pwV33tQ$sM;NbC< z+?YmP5H)oHVd>6N6ABFj<5sqMui@>*M8d&OTBbsh3F1qQF&)>@&)3XmTkjK$@o`+) z8(4Z3uJ4AO1w(e;e(Z<%ulnks&P>@zK zp3=^QwVjX(=Hf`ouT*8nN=Je&CnCsBvZ~La>N^XQ7}*ocT<$ZXE=pDuR9jN;Q-w@P^yo4;>&O%q&0dZVez6}X&5vS#+Q83B8Jv7b z9G9FnO)zEmKerh>_in`Tb8Nilq-UKF7cA{;1u%p+?P;?CJX6+&_dXAAUjvNE7XC zicjNKuz2oVyg;$iqQ@ROa|RyUxDifkGj60fQ`x&)XbhEnRv``z<4r+?12Qhs0i;$D zj>m1+0Y~)CPMh6pyIuBdXFV$PqWfl$ud{mjz z^a;`H#p%N7z*uGk`9elKO4{055T?J+=Lh5_;ZYJ3C)vNe%&oR8jOlf&?6scIlJixn z-bd*(=A>&8RFudDD`jC7z?cyqlikVK*swtD0;2Hiwb+J6z^JO6M>S=C>FVkPp%(X& z2qp3b1c@~~7!zp~$M6h-TnTuwL|mlH`AG5Nzg!-Zai0}ZqoCaAAcluVF>UHhN}P?@ zGq4X%qKs5$6h-dT_#*KZyN2gz;f@Q=ZrySibzBIwS8@E&mm1@kI(RFNi6o%fCVw1& zT!@fHKqnp#|L2Hjg}P5%=`__jxBUd`>czOvt~bqb7nr5@FCHz~>wtW+om*RIn`Uj~ zBiF5I>0O0_kvckQVH{Yht{>6hfrHPk4IdKkjj|4u#&9R1Uc0qHqj$iiXtMR=O>ZGz zlVX=KFU!i%UsNVPs~fMmcL&y1QM)#e7Lx>bQe@A7nR(- z6{vCP4^$KOzlY*x@yM93Yc3QO!PP(_oI-A}gyDT7!u8?=S7unc!b6X3!}%)~h`!x(uk>Q=<{dbG@e~}=Wn%AO5qGbA8P|Sj zhPa%%ZXC~U+lOkA_4Gn-ddyX&0(;7g88~d#bhJ_oS=`i!LjwkW(bbNJUfP7xvum(o z=@P8(9mFF~KaGxQlX3ZxM^ZvHM2K}ayP#q=wr}2w!sw_>R{r$P#7`tvU0Z6ts`)hy z)F$yBY6Z3B8n=1;F$AT+>b!-bAf_zWgJY^+ssjNstb4?S#-PU9^B76LHWO)FBZ{V$ zN%YrIWOI8^$c-QzO`xf{Q>rWe6^*DxI^PfMX;P_jCs9(ukzjaWNWKr>1U&vKRoJg2 zscvL(?5hX?FAz}Qo8{K??CO{sP>3UF)DhomyucOchar+7W-wXO{0ElQ#WSu_6G{OXs8`0|DO*kF#U zkKqFV;ElozmB?8&L6*&q>s!h?b-NZrBk$@1osY&E&pEG+tqvT}XbQLY%?~WMy-|;+ zFXecnp9#{EVJM{t0k%7a-=)>oKwfax5N^*SM6N%5p7fdH1i{dG9z9!Ttviq|h#t%; z5@p$)l*K2`>;XkQiqml1Q0?3cIzP&y^%pB^j? z1)5M}P6z4X3?_Fq;1ic0gX?dyux{gSG&h74Z%QEHj*yuhL+w1P`7tc*GVqD_orJGk z_Xs7JF(qST8k!65`b-p#QBitMChy?p+aJV5?>Gis?ag@X&JCE|n!pi+A%nUisUp@i!;ZmA=`{%Jp^n$KDq)1;G%4isvKpnj=kyz)vgyrpTK z)`Jk!C|GkPYPiaTiUYDjKAB1;Lo~8X(Y_kXr0KsUb-^&oDx@HiOzPrMQF~j5>_)P= zEOjL#C>HXF#~LUZ#F3@8j2C_ubtvpLAv+j4S!|fZa9X~p+2~3n{;H1VFjO!$IwH7! z;ctl4x9uZfc-+U`LYe}5sW>J~e^c)Wm_^S>Fun8m$*BU;rBDTuZrU>ZKhWMVvj{g}b6B@)B%gROtL>*MHpfB@7W#uJ4F z*@@Kok|;efV|>yv6+u?Tepl@T95q&Q#qGrwG?;vU8E=hGjHd3$IEjdoFLmXSNJ6^! zF#S17zmJ9!0$yRENk0C39kCSAgpGrXal}gn$#;C@C%dLrvn~dyBeANcbk?q4Etsh7 z&ub=Jf6N5OuYo9-MCh7dy>bv+U!lqCQMX#RfNcl2WW zo<1BuCyGzM{}fDaX+kDb)YpsGwW)VvOX{Fmrb4w5Yxh)f-OcyO=SY|pT(V-3>}2m+ z`8003=MQL#m+`g}=g~cCBS!Z`oX$^>zBo&$q=F7o$q=4;X*XOd2%}UeKm6<-r0Da6 zgC^olU1)6Sz`{ePAP`l&yEhZQ`DzQsu3#91or4u>|<&wF!;Nrg?eF}x4URdEb2M-Q-FA3wh8 zXmLi{)9})0K}&gBcvZc0#C0%H1p{`qnt|t8?)dJU3X+bxq-FXa-F53tq_hRJfFoRt z2e)dN>VcZF#>7i#MS`sFQ0mcUwcpn(ZTs)&bzQEivOQ2v0TU_7X|sjk5gyEbxB!YT{C&X@l5Cd=mpWCRtoY%PbQfG8gNa$+8ftwAN~ zyRzDLnX34qJ!wRc(BPYDP1w@($gpo-_=%4{bIb*tKsE zS*ozvNe>%T}HSNNUP590; zPhwU>jJl?oavpicam9=oXcd5nixxW9lc&ytLB;ME7v?PfNcZB8)~>;DZ$CoGhBu8Z z3;3Nzt?cq?TQ#W$==rWO4sj!Mc0)-ya|5EFj%78gmMb`{33`g4=u#p{XGf`>%psmk z3U{PjvBeROV@!qGElwJ(tyActPoY>G#n4a>iqvti&|zak2gY(a$*7iQ`0@?Hqpf_F zx{`u=oRUuo$!LXlBixd+O0Z?x$np5iP%vHA_2LeSf6v`YAx|ei-7hzCm6BR85)qA| zEWLJ9W2Yn&VPdn%7u~Fm4%#9cL&-ueOkO_H>+<&*%N8loS!iyVjy=2A!_E)N65o>N zy28c}g__VcB_?PyP86YJMCAf1M9b8gwjZ(xhn;aUjA)D!a7hxfs~z;bQHXJX@~k&X zsju_AV5I__9IMx@$C{l($Ud_NCoY_VAOHQu z*wQnCTOL?L_s2e*ar8Vqzcr1Qc6H;pLpnGk;n64dA=1Ff#>Cyv4bpKNq;4mL-D4&E z@UE8yxyOreo@^v2;fda0y;!F^5ftHd(g{yMSgbah!lzd~ji0u(i%2Qe=aNYHXOzz0 zaJht}r#Y*^M&H}86??aCkwoHi3E#|^OJ?8R-ozhoX(**0OVlo4OK;&#g|_jlQ&-E1 zXk9GF@fot-pqw%PytByW3Sw*9(%dY8mAMG2vQZa0Bd(Hk;V`RwXt;+0`H;YYn^K+1 z>w^-KRJW|cs!I$WmlbjuaqA0()fgzi%Xqq5*z#AA#B8~c7zO4*R+}4|#g|2J^7PsR zl(@>JtU~G0QRh)th7w7oqKJ|V!L=;Sz2TddHM&gdNGc3Rj*GZuFc6dRjVRPQ6?%RF z&CQe1I%$?lu6)yKvLEYW1;Yutu-ef%buOF$cQ^FD;W%=oDq5z^z;WlDjYtd2sZ@Pk zK*Q%wc;~>tYh8vaxzx^=*G3pVf26kDb~Kb%;HI_;G`xHju7o#QaiIxe@D*g`d9JNf zgW)F@?*ok&39Qqua@-oB&Gkk{RUh?a)b`3XxM}?`WZVeK6MeLO7_=n;eIkLv@o)tn zx~jjQ$E>m)cicTT5!<}JKU9xZ>H&`9)`?TP6HBa2NKmw-;PLK9*RZrVd{7dF>_b=@ zC=iu`BtX|3cO%@XGjPJ5VD|0$16JMT=>4Q}8x4&=7QSwlIv#50SXQS+Xo+V{C)D!Y-O1rB)W@qQ>?&-Vx zydg98uDXt}|QVxKq!L4*Ne8M=7i{OBW(f}!&&bqma6&|Pm-VX0n%W3@7zu2(qf`4)s^?f_O@lbrthF=tdW&Gabm<4)=Hv zN;z*ttk?onFqqJxOJ40E1rm1Lqk;My_j)6?)*Z0X6Tvd+v9;QYKt@Tr1S2t<{Ah>N z>T!^vzA)kR=`|A+l3ouFZSlsL@~~g?>m%VflCfW$Z*ss13JNQOHf(J_kB=Q;=2L-6 zy~YRW1P}FEt$`KPM6|z1m=P-_$_Pje83-;(!%5t1&bBpse|lay9|3_pPk(2?qjcfy8l?YJn=R z0-a!GE#%x2-%JKo6?@ICI8}8yz=s`hr?`QMShB}U=e8^4lVjDsTNt*M9HL}#>tq#97ws9o8LepMJRqgwtcAV;5j{)wEA|M13+l^ zE4_b7{zPrSdVd_^8*dK%m$9-x+@d&ipL3N*;*gX8;i0>E=?3(72Cy`@$cMA>(lHLI zt$bLb*BDvZ8l&`!Xx_-Z(7=PQj+J_eiOK<2n6<_dY;^nZ%fJ0mxc82e@XjxNEAQne zFK)o6pS*yHS(Ou4Ie-jQ7O+COumQjPzdr;&_PyT%P2BVNZt#3?twT(52fOf>-*gP# z{?IA-r$75JeDu>7;pEBFcnFW+2e3+c&wHPNfB8$l1K<3m%ka;C@Nd8$z3wj9yxszJ zFTjtz^NsMiC!T`+-V_Gf@@|aDfGj*U)Fi?>FSnZseNM%>1$buXI{fH+-w*G&`z~l( zp3W}b$2Ty@(Q7EL5*i?|I$ zSb3)H;h|3-?2z2}q5+qzB`onOl{(?P#Nen@6sRt#{}S95Z@v;9oOJN^dL4WY($lJO z#X;hU)JCJ0RJl~)a;>}HhT*u!uu6IiWpZU2o|A~;#8}g&0&`Kw&5(wi{lOS-(k8n> z#=MCp(Av4d1jA|#uScYLV^CYESZea~@quZ*>qnCTzK(sy>r*O30dYL+LsXx#C2OU; z2>nR~_WK>!AL9Gy53#IVgXYpY&sa!@tXft0{@VP03Z;4cei2sJ&cgFoF2LHc+u-bj z=ODmLrq>^cTm`f7Op5rWvzaj|(%RF}HFJ!!6S5u@$xR~zbvIac((TGJf}Krxwxg>c zp@q-^G!@;hSeV?zuw~cjAoOz~`VvrwP93~)s+Vpu|8G1g&&*{0&S1;RMaGdA&UCFV zrL)k^&BiLRG`HCDa&mut-w~RkBKCn5p>=iA^jm;3B@qv8-0yaFp0zDj9+89uq&flqw+ajz`l=b}#5idyW% zsVi5Kf#%B)-@Lf%(saWKVNEyBK2g$Ui##W!> ziIGE^!VZf_hA>GQ{5_s=#yu zQBZPi3-aseledW?cfxu6qsSG-JZ3Bb6A;PQoDgFjAnsrv4|;YaqYHz9i~>{@-wU>d6Rh$3RkEoWmeFw7wlC(V%iwVak>?HYrq=tsr9ixY(*nnnb)< zW?Zg+?bsvAA@smI!GV=iF~-fwltm3`Z*~=1ZxTWA+2^tfy>78X{S&{V^adSXrvNNQt@vWMHPiGG_VegA&u+CKLDq(UP z$U=f`B^H}R309{d1MXUj^IuZ9snpy>hUo%U&{i9d;jd<mR^Y5FOg{4 zx!3P>LKVw+LK*khcCQcr@t58Y@jKoKU-Q=c;a%VHAWJNLgCJfae#EqYW!%KHw)XBhBdz6mr#S3vtds3J`d{knf)o1#YKBbu{1sXt|3+!L`bYxn*s&I=OZ!{>ltBHG9C{Y z{Wa?KpugX;(&}UG-BXK=jHn`Nw-u}p1VL0MQ!jl%*;&Y%UkYDuAG4P(%+DX^Sqbr4 z2-9Dx;CrYoWA$+k8qIm=ciSxM(#0S)CP*^|ro#Lj-1&xka3hc(%)s-Kxd&~Wu%Yr| zjb0WdIYoR@sbWi{1h53kDfd=2c*JXRX!=7=WqRdK(9=JsBS%t}Ox7b4(o`Fl?bHlBIi;X%Nx{Y|6OTY(AB_?`4|ziqd>&~A^Q(-}jD2!*Bq*E$t= z_kaEnyyv}-!%9QJ_kZVo@b(AqfIZC4cUl8zbw;q;8DMfc%%IECJenFNr_Ws*!oT~) z--UaRPv8f?>pr;eL=2mq2%q~=n6xj#PyX!(;Qn*-@a|vuzwr5sd$76J!981sPd_t( zUwqF8;qr|(1ZK?1C0`XQ#{7!JBDEyamB0f<=rTV=Sn zv={2f7NNGbVs#U3n&|jS*5HoY-PHCUP8n@UE~oKicp#h^``IR6*?H>`ZLZ)T%s%~L zo3!ocY;me)A}KJMgkZFUOd6BG9A@jx)nH3kk-DH@K?iwgB_nn_d(iK-F>7DOyh6c< z{0i{ko(@EQp;B)!Dr-0$@TO5D0l>ZNm}eYQ{1B;HaRy36hOA=1;K4qV^8%Q!FR|Mq z6BP>qyZOJ=CbGQfb()g9;lzL^i3LN~LZp z$_&LKH{#h(#0FVY+gif+7YEi?K41hBqlIj6V$&ieq}Fus;^%e}^8)co^sl%AoZ4B7 zkP)fH#w5i+`B{qb-tge70SX8cCX-se3v*-y66rC^nVP}1qLkkd!mqI2C9E$tTiv9S zK;^HcvDaBUIJJi)C%cB&{;~6}H&YL2m{^G%AS>uVTQ3)p)(--wUSMqDqdd1jrA`-P zjj|kSEQ|0QCP4}R)8tQ@}x_ug|F zzWW`og<4g?RXpGc^54gV^z70I;zP*TKJbML@P8iPfX6@gyRfp*gv+}gM>nzU|B33Qyx+`Plm&fv_yZYHmeOL`atcT|gpT_`LVscQ1VN9qX`xAN0ddJ_(yo zU9^cy%m~)7Od_=tw!09)J3Qlx@);jBVhNR+#o7m{d5M`Javf5PrLs-3jifJQb%I>! z%UVfRbBGRGx}c$@?eBN?nd}&|m%G)0qSqc8*zfLh@3>aMi$We0ETvuWOilW5`ojqx z*h6R(tNhr=jQBm!FrE&y5mpN2QXQ5SS9z0BA9li~Sp`OkP%CNB-)Ev?H0fJ3(3AsK zt+5Kt<}$x7YQfQOC3*W19-?$m2?;M?c(GFA7gDQNp|iII*REZLd+&b$nvEv(MxsB~ z?(D)4gW}xcaVS*3fSDdXFJ>Jhjj#Vi38v*K5Bhmy5->oT(j3?*pp-1q4y?IVLN<(S zz_Bn}lOCXgcSEH2MpE$FvCC)3b7rP}b^ux9+dq-m2QauH_L;ha_%iD}Fo|emY`m)> zzt-3m2XR7}^Jp>Ol5?06_fJ9k@`r~Ot!)x z#p99HqSHbkREO;a<(cBiEZ_J_L+jZFN77c1u}3n$5- zKMQc_`AxX}%t`$B7@obd%X7<{UIDfe2;lqA-2uN)AWgCb_-m(@;QQYHB$V+yD^?58 z-)_MIK4<63Wq4D+2WPP|x%~PEVSj5A20J@^#;u+@1@+}cxc1xy*x%TLuYdC!;UC?1 z27d42^H4D*s1cML&*0@#=V0O38eG3}3A)!ept3Lzi)W5O0R!^Z<{k`ocR&@6HgXHq zEXrbrvyhg$II6$NL_m@jFO&q6PAaxq25y9&Vz)*18d1>gi31@SVfDIL8sY~W;#X^T z6B^YfR}rM$#aTOEoMO3=2=CeKI~MH`A;^&fAi2ktL!QxeF-p_Wo{X&MXr(Hr%C^Fb znDGl^JeUS7q^Jol+-t$GyMs} zN&;&&k4OUyBROIHtBo?;*uDhooiosAF2ZQ1&G}ytOfb7?!@|NkUaK062NPI3xd5l} z28?PLOh;ppDG7wTBa$D$;&l9b!XlF|xf(Gl1*auhlt|PA0PI}B_-X@ebrUFe=sG|3 zgSuS@>Q7^5HeRkhL2S_>!Ax~yJS0=ufIyzthOpAuf zY@M`Sv1RTELd;3VXMqKr6s0^r9J8LF@e$s4#BG{}vbTsZtJ#+&NUBtG_*7(erm7D_ z8VjM0r9DPf=9pTRG#kX<*V*sF#?u=x#B6-7GRJ5xnmdp?X)HapBw0!6r77dQPGX;4 zC@c6VF!0#ZjE-)-875jOl_29PGe=OBphM*$CtnPoE@_yU5EDK8efT~~aS=++@JYde ztDv>iDYTEq;~uOZKLPjOc>?;KXW$KYFTs6xzXqQC+<6Qp8kUb8hv9G+7MpY2)(vrQ zc5&}6%wwWkU&cTN@R5&y0peaC1Me=Jy=@83uHV7nptYqEey#8xTE71zoH=_NJn{J} z5MDL1AGEs3HK6M>f=;^y|M=_~xPWKS-}}wqht{<%_`bjTP4GOvy~jTMIQ-C;Jp}*e z{av`;ZNZyP-v-C-It!n;?=JYCzyDsiYk3vE=k>2+1(6RNUx7y+{zLel+fTuNx_A+O zmLe?O zxvRq|&a)asw=8y$?iXu6oF-e{sTKF55?{ppLN>G^jt8~4sG=AQ`xrIJt3<=@?p4@t zU*}THNimGaB*k8IqW8>{r@^^Mco@(ArsuEK8!X_+0(K+JCdxJ3cxB%stW=u#xDwyL zeH(13f+8)fVQ*Jl9fw_E+g55&t~BuVRrD0F>5zDm@zMmv4 z*xNl8jJYvA*xI^b?NbY^S+u%xOlBA~CmZ9&t5@Kjw>$`yd`pY9<8N)Rww5 z`%z(FA-0LPh|$X1zB=JVpRJp;9#wY0=8!Q;9g@zvgHR>Lt*-wqnf}yzv1>nX(|*3@ zq|v3t5xaTDG_qgffaiL^uA%~4Ie=4B)!I5R26B*vA21>4GeB_j1!STLTZq-tg> z`dNU1N@-@&-pKk9tYUKBG<$@#e1e(am4H$$5h z2m-Z;iDnyFE%B~&+)bdP-k<$w{?cG}x*!_IC8?!My$y!o~3 zu)bvAu2W^WYrO{doNmC%TnMe5>+E()mJ&p;K6j=G_ncjT)h0>sPdPxX-*y`&$?`dY zDX`>HtKElAcfghseN2#7mYQsDJ~`qVnv}FRsSgy1go*!YpCI8CBJM`8j#*Njwhf+J zYq8{eqe=z-?WHU5_kZEP!PjGuJ$?2peEZ3D7~(nm{N^V7t+@u=d8`3LJ%$VIHeWAx z>&0vFu6th#zjyT-e8PJVI49-3oJrG~#kZag@k*7^zBV(-Q=iT1Dkbku$1xdk8hWIs5$`*xbCp=1^R{ zkOSi=;=Usd(6z2vqJDun8KX{D23UA-s7QD!jd~L=I6l{~&CZHzD>Xh?h7-fQmTILU zsE(<8cvO9evtbKA3w zYPrS^VvWesm{gl7nnW3vvPTV>MCQgZomueG28IMl1et*=w;i;GG;a-?HOjKC7z0@tcmHlT?gJXMpF5K&dHtiKDt3P$@V*t)a@ z{Tn^Z{>!|#3D__+v>zpa6R(4wCkT#KTQ+vCrc=ghPD20Y6g%Fq1a~Nq;aEv!Do2?*k@kEEK6hQ=xly+8Mg=2$Kf4_o z@b__lcHs2MHMsEHH8^%`5q9uAxVCYPgF$x~!~9&8+mMg{(dV$DS;Ay^2pz0S?mD*$ zg&LkaHvrCGdJZ0V;BMI2?!%K9nAg{DgN@IehqG9j^m-8tdIRC^ydbhIfThM9{MwVx zz;S%7@A|sG1V8<|55xXm2kNJ8gT~ql&kpH!t`PVLgH-+03I3dPe*l$69rnu=_#g&` zM?U`yT-?}#aIXbleeZp6ai_zSjL}{j6jncz1_rs~C!s=unUwhBx!cG6*U=Iv3TY0B zIke4{SgjhBbT-^*FYP4fH<2YNMYey+9EZt-4?Yo1)OKJTT2f++nU!RBBo*K5?{icn zc@1g@jrMOqsc`}t_{PQWBWBmdN_C#is3`NGytRf^Lw9!%G_|!#C9V``phRUq@m8p? zp@Y1PWjVo9B{#x@gS^`f^;sjVc4|B@r_Hq6+Gg#qNpFugYQSQS5lgAe*X9}COrGn5 zL5C3!WG6i7wmExaUk9vqnhPhGu}?lE2LD~fzZa$$^rlj=O)xN$&pbWPcof2Xa|xe+ z01tfSTfi)opo>{xkWhzhb}H?ph+8b8uuFU$W!p!Nr4U;w28RwYpsjj=9D+VGh@B9m z&h)wCbYkF9I64a_HL3Z4Rb6!`JYlxWy)cRMVV+PW4-MuxyjVJ04rFyW{&0D!qL`lz{yN@K40IVa?X zXkVkin@#q*9GD>{L_}vB49011S?&un!o3@g9ft&VpR&Llw_w0}fYAGhNtl#j*qHpV z*;_&qi}@)s{iW(~%HWd7!@2E1O>81UkyzQgf|bgRG0YX_aIaK2d#Aug zqJ$(rL{*`f!C`V0#?}YPxb($WB%FL$2^DV5Sy?4zqNM6IN6B&+7(p36a$e@ChOb)| z9_s}-XBhRazvATP535X+*~hb`3YC(`+!J_ab8igWpLiCpkt88cUAxFAx~DgKP z2cku&I{ewunRotjPaG{93vg+(3(x-c$Cw&0#>92w(a-YrA$QhipWk8S@~hj=kUS&L zkOQ4c)Rm};IITk`UK&RwXpJ>|6tDjifAARm_NkL_??Vs3uRZY@_*du7z_)z$SHdQ4 z^}%3(0j352;LK@={>nGN2e#Yr`7fM@U*Fq??_6DmuQ_=VKCstk@0Ne|(LaKJ@l9V3 z_kPveSZ?QMKK~^A)W<#!@BWr=hVOdEH^F23eYo({W%%;5>(HS}1YgrOe)gjY-K61> z^TsfnZYBdXSCU31%Cs_^TC9+XM2^8_9fd+{vmm)gwar{7qY?Wj%+-auPf*=*DZ~t5 z2R5!e&F!1hr|*G!y^jB_#r6CQ0I?F*nbH&~_UG&alq` zg!m|964J%H{sHqix4E(jwFW?2_tc@v{46S`azzIn@>z{p&^fps7?;~SO21(`T6^B`p z#>-1u)SF=dgEHEx15EwnxnS|?w?Rgj(?dd??@45<4?D*&E;*q47WZGJScUElJZyFV z7VF1&KB6tcj_C!Q=0JM9R}!aGT-l8C_?@h!~WV@`+~cwR_S8Xw7+ zQMW|Q%>IP)eu-1)MQVeyIAsz1y-NW$j>DuA;7P0Y-Y5woYw8n)A|tFOq!kvF>>N+H zT0Ao?Dw}X6$jh7ZHUiPU`{73(hEY(2V160i`CIRU(qa>@o8}_)_u;e;y+I4=)iMW`=_G#MSLwS7?Kh(`LtnR z84Nr4n)-G_S7C1cI95w_EOFb=+P%RpkR+{NDwpuF6AXwAZmEqXUAX+*6Rh1ccltG; z<`f4Kn>NZ#Y#H5bDrMQ@;Gw|A`boqI1UF-~3z-ALfX?s{oD-t5L+hj^q7#u%Uvb(a zCr%~xbalqDrcX4FKGTT~PKY9CEc;uWTZN`w%Z%fO&*hi-%^JR~R+&-?s zur-Aqrnt@W5+^^#{T>=rW7SjCqHI9>(6Y|ttWXIq$^kpvOeNs#+j;_-D0%I-qz{q+ z(q9Zc`-Nw|$qhFvIFq36oFqY(CxW@bSF?zP~9H z>kY0hhIl?cF_BIZ(Hx4+RoKU~irgv5jfky$>Ab;;N8`EK9S*JLV32C@&Q>NQA7O{9 z)aD^_APFY2+y)-fp(;oe)(0VBe$#^XPH zn__Y4BqLR>Y+hg)`dXzZR=#u-)65w!&}i6W+&h`kw0dpn9c$vG&;YH-7l2-45ra@< zlxHtfn~cbkbU>Af1xwQK_~DFLCWE|3#*Aj5#}>jGRBNlSx_XjZC*97Tuy9%U5Dygg zY!?nlt$7?)@pV&c@A@W|$NL*lTUf%^G=Uw=3d;Dw&Mz$Tb=tUi9;W?WIQ6kXtV{J9QH3I>mzb;4tG4JY$We9E)9>1Oenc-Hp+nEgpN3 z#FR#yF`jhjvt}c~m;x}_77$!LENGr>74Y5~sg&54nkC&N2LjzEH2kI+nzE!=ofbuO zbBPqWGcCQJz_%^&P&q~7Bu45qBVU76hj!FMew$c81VxB9olc~zi3IzG( z_yLxmL+w7iu7seR?km{e=s`KCdC^t2QgCw~ZdZ}!9A3Mzt&T0cn%Y*{scBgSVo7)` zygf+iuUYu343lnYpdB6g#SZRnp9{n@>Ll^@SB80zxpFGJnYwCAm9ZtHVZb zfWM~(?W>mTOxC+h504!lL|ZBWbi7n$ugspu6xYz zMdFS`R(if*WvJ<#qWY;R)m6@3vh>21plLZ-MMl7Qeg5ar% zdqOQOVBUb3dC~n}YBXVW`6SdZQ`^SqKAjG6=BXYU}YmrPYCcma}wDMor3VkvXZCK&4LG z;H3NA{pAj}Sne@$MvathtJc5YC*9v+r2@6E4Aa7tH4Z7^rP*$!k7q9VFA;BUis$pS8!+76fpt9JTbR_d zk~$^ztr3Ly_xW-WHm7}PZEV0~)c1DSa0tNUrrKPVHN}KK%KnA}F?f=8N$lGLv9g0l z$B2$)phjxFbfXCOLtA%6Vd_Q`!;n4F&Y}aKf({MM+4O4AXTsjx!bzBGtiX*ejEI9a z{=LZ?fwJ(BvnXJ%bHYDQ@WK@;4bH&YcySh+P2qSH$TpXt#C6vAa%^dW{0*mapae>U z5c?F=CL~ot2$88pnBavi&qHl)6)TD|Y-4cP-*032UtnB$gh6FAZ1Dg-EYzSme;bsr z%-q^0WYz|&4mamkV2szU)!Kq`Z5gWgT6+6kxW0P@PQKwBoVe!<^f4g0tWBJHRCn44#Oe`(PB5wgQfU{G8)D;K36b zdw@E(&B*$JxXjpkmS`KW9BD17s#2=NwqGt|6YK8`VPYxmW;x1OK7&O%`GIV504k3{ zBl#OEOc{&k3TdeU!^}gkla=zN%*tC@Y{HN~y9p~^QV0~en&?E1&z;dz}N*{1_PeyI*h8VNsd9uBYt2peOgr1C~eZX35+H0UKu`$pB zu@=KR(atj?4V#yCAnF^auz6cnYsc9Z7?%lyMQ4Q~X~V-}V_StLvcD5jGdi%6HUwCi zFt?0gy8+21F=$3${w@ImuW|sX2A3oS4i&^+IfAvYb=i!%l}hA)Zl;O#(&jGzA9Nb| zL=yXES9#n7;^h;p&p-VsFRjwPdYu#8Pwae#KQm&nQ_^*HdyBzB7p`98@9&NVg2FA7 zxdZjgrRVXSE%Kfgj7ugY->6qHHGPeNFBx=D$Y!&2VUL}`}cqoK9~wZND1 zEU$ovIqNEsH##@cF{2-iU@+*3-72B3*oASz407T{jK*F3zymn8b|;tbt@fscrv|LK zHNr}SXB;%Kzym!X7e+jw!xE_u7ofFw1I8y#3VH{B{{S}xaXpHKsuyviRt~ws@j;Bw zGnx!phbyvLQYsdK!LU+?>fCYmYwzRtw|2LoQqZDZLsb_AAnIEZS*pGSwOW(kN4tF$ zA2Z5*x-^ws4#8cwB7fIgMwIN-QBg5v6*fHbOVKNr0Nuy;xB z8C@agbUbh>_}|O;S}_@-#j?1l+oEGNcRv6D5Z~ zySgNnkm(I^BDlHeOoHS{2)mj&ATcYj7x)n*hr*WTq_rwHM!73#z2vC;K z)rw0V-~go(q{GPVKA6yEE0@Yp#gdm0FEopYOzeBNJiKAM(a=Vz|Z0Iv#`*tT1IrUtniUCW5lcfU$@n<43JY`;B&^5D&XOtfZR@ zY}O`8QG&qBiuGlKzCf6jf9}X4BN?tuh^#oWQu#5kK8Ph3Dhmy!zq3;$k<8ff&_crL zx^b)4%Ak<62)zcnIFUnbS=A>a@6c6pbg1Zl46A`80-z)Q*);J|Os}R+o`CuZAU^a{ zNZYzA9VawIZL3-{QxDtZf@goz*g|wk;sme5m@29QOjQ~Gtl+gOVDKn{ELGQdGtz-Qoz`V$!UvaJ9F|-$Q?7O}EHHU8!mkp$ zFqT=&aYU-ZSiz7F{czO6_YhktjaaI*Xvi0>RG!Dz8AEsfI*dmhh9i@^Vgug;>4;IV zD%bF^sg+>v^dc;tU4ikm&pl(6GC9gfnS10=a50FG`aqy`BHdzPt5o?(Kn9+fNjqvPvys?jKr60F@SFsb zSo%7>ErygcEYwPyYvINyi3S!9ZINaa)a)pplN_y5p)=*`$H6jS$StAv=ox6XHpRrH z*T;ivD4Zv-Qv2HSJeO=buqqN&X9_DLd-&%hG=uZ z@;qGregL8aO~RNZOghS3!Nl|@RWzi@hxe#aMNXJ_=udMQ3{ZxhDntHRW!t)fGK{67 zGtz3~e;Z|6kB~*^{RLkJ(yi^3F`XTOICQ0x`Y~4IH74@vc*cH z_bHIfLQ_;w9_(n_T!taJJ5A}M^9(mjm_c&L;ixkMH^_W8 zv!$l8`cuXaa*ZV|icVE%+d4r;z|*y}xZ>2=*8nrt%}y-RRUibx4tUlUGrO8+6x^no z*kGc(1Uuen`1;Wit!zd}#oQ^Cs(^^QBJcs+G~;cUUg~hd zRKR;8aNg!70sCq%aRU6B(!+CQ{~NusFit_6NfE%uq1YqJjBR^xWv$b8XJCn|DFAqi z`!m&PV>71=5G3};skFq;VT^h~%u+^p-yEK`n?q1b85ht@79JdY4t^U2P#L(jWybTp zM`{s$aHJ12wJHH%s}zT3Bt>QnHU=x5NLMic;eDzxdBDnMLO>k)5ZOXP3m=VIKP<)K zW*Ejx$R`t=bbHp^@=<-ue(p!wmnl#$p*Bh!S5=CGq_UNTt&F`b5=r(@tCS>=h9j1~ zkB4L4Gls??`0pB?r**PHpGa`#L}fG<1z5(?Tl{?-C#?l0AQ|zipf(s~_tY|=fjVbS zD)QoUh0w68BF#H!a6XX6TSz|}{+UWMJ(Camrshf-L9a;XU24w+nyV19l9AHOey_t2@_Mlu($>|al|#ywp~UN^Tf)h)+kkgzGY)2%^fMz$(sx0 zI|XTV8FdUcJES89Z`zSaUyB!pw5~`nvc$nE5(y9!r{p`>*p@4v*tgp3K-acA*!PB@ z;dEHxi)wPd8f8Qg6IM%{B5Nkc?|3kh966l?SnnS>^P)o!NPno>Y@JnqDL^`SvPzsT zxC@}1OQR$Skdl9iQ|48Nb!`m5fxJ2`Flqi2Gdh5mhP6^J>hL@4CmhK$0F$cVjiy## zOmsmntSRR|Ryoj@dwA-G^_qjs)D-0&VBTCHEmBrx4taJH2jn7&2?t5aYf;#sKmF&d zX2WY*B_clb!_V46Sfu03_tl}Z2x(`+5i`~_zX}v&{!F*PeEXY`tK^F^urro1n*m1T z=t_ZNM8RPq{ztfX19HBdP|%2>fj?Kr|6ZWP3XGle1zi0}n@d@l1JdBZjJ?|(vZ$du zpd`Du5)9(RW7*=?G9?gaMIwh48I~(%I8!wd<}CeMY7t?e==OG5p>u>Uo@NsD`kW`0 zQR`}eZ>Ui(vHS>t_`4GXxDdo#l9NE$}EB3TlW9m!{$;ZSXx2 z#-Cuq0qGWTmOp}CyA5%%24^0+7Z$K8>0zLy8BuUZAknst)b_@4#!+q%$EYXmpqI|a zLU~CEWtDHUwzCOp%SnN{sJF(gsk&SbmHx6I|xo8nT*`XX|Hq|>}s;; zCB6lWW_qRpX-?1vJ+WI8s?`wPI7y=ogv=$5ZKjL#HyJ2GzJaY8 zRnk4Ht1sz#ALsV(Mst`e%xao4Qc|oqfbMG=L*C(|7pJ(F(L~Mt?WTF_Y%F0gtYQHM zy!=`N`NBTD5#D=nPPLb7${*6?3gKHe70}VtVYr85GyodDLm$R5mU#hPGWoY~>Wxdm zOJSyUDWvDv)vT9h3J!6r_;2hJ^IAY>n^NsvN zSsBTtgI0H}7l22hf2ih;59qv&go0z6uWYk>A4aH_t^`SV*DIP?VU7KBF}m0d+QvD# z!0G}v$!us<-|=)bGLqgi6P%;Oo?M+q^JFRJ#Z4vcom<$z-R98V7`p@|Co7*qB^xN3 z5EvYQL!;(DVU+l(h)>hW{(c<3%i zN(PPo<7(2zLALPD;Pd@0=@-Vvh#O75~bNfi|`!*Aa3Pu)BNW?hI_?(Ap z4?LF$X*~Ja6b@2+Xfs#+=w@t~+=GC;bs@Ka&wMDp_Vhl{%+HQNoS()fqT&~Eh8!pq zYRAGh0c=obx8@!e9W7L1-d#1y+kKas!CVduu@Kq6)2;(8OEb$^`{X=-@(iQ{9nX-p z3cBwt3eb;ko%d>X%?W|VvTMz;GYf(9Nbg_~wMo|b*TM-2TJK7iB$5PZX#{uaxzq*ZI z)T-5~MOSU_&~}-2_cquqW(Vw3*}Q!sQu7xZW6y6CclZ4nG}le86@Ju+MRI5t8eqCo ze9;dP&XY*kOy4t_>I%519M6|XDZ*TTE1;MZ7Bh}`!cqfGye!8xXkD;WjUX}e(N86W0D}L)e!|vD+o{^f>duWas^>A@4NRl^A0`iVK}h~ z(CgX*;|LX&_jb=X$5vse8LYdt;sdkZ0CtDsXu*LPFaWaXGqgA=j0G!s6n}t>un%id|niW`eXmqzjH~qLo9meP)!k) zyA8{YhqUR%?Ozyr;_1KCtjCp_#;zR)BF5%F2c@oD)G9?{W6uMG8Pi9XVkSFRaaQ1Q zDja!NOpQJCl0c|x*dGf$i(ie>CvXD-lzWhIcgOqHA@ib5n%d{qNSpK@@ZYP#0}FlFOZ2qogqFVbbDl&pcr7%CV1aAOr5Q{b%A$e?ji02 zac^@>X?}|d%3tL?vb~ZwTchi^tmxYMjDPzJ;vCQ7xQuQnuE}$Ts5>HrxU}eIqgc3x zC!6=;QX9K58%~2NKjBaoZM&=kD%O*+)O*b>X3j|TKKp=v2$grU{e!++^R!)sZ_D4thS)8yjgd& zt0!xqKQiRdi4L~4$e(g5|KhyeMiG9pJam6GOiJ6Q8O(CL5D`O@5=}p#TI0UDVUl?S z+-V0OS%sb7^pBwS%1gAW%L#M41aj1hqAS?|3V9KE@tiK(Ee*tUwMI&T2;$1|HFf^$ z-#6sC?N(lWFaEoU(2KZPep65)Kvq<_X9 zY$p?R_rjUDA3PELiufqsGt!8ZZfUlVZ>1-~5^2)K`QIsmp~K2XXNt(k?-rw=!wCE- zmxJVw>}_ZYqhsvU&>`&v{}d7Z9c-5-eX5DThVM}MuKOcR`iIpY*+0N6csVNYtOE~( vk;u9JYhs_8i(PSN7So&su08u|6~oN*WUypab!gFw>13I~Ew9%bx!wCeFNu>& literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/buffer/buffer_featured.png b/content/ko/case-studies/buffer/buffer_featured.png new file mode 100644 index 0000000000000000000000000000000000000000..cd6aaba4ca6a0327e3fdc9f43385b4199ff3fccb GIT binary patch literal 5263 zcmcIocUTk6)<;1F5kZvR0-_)-p(aE^ganWhx`_A|M^4 z7b(&^Lg-C80#XIOWan4L-i=|#fg9Pwn372ejKh!9vUs}ca&TO$N+%VHoH5(;l; zf79C)Z{V$Mi1T*D$y*B`6+sGKa6kY7PsW102u?&dxEDg;ce!xj{dgEG0Q#MT?1&Kf z!zg2nE(qo9iU-L`$cf`5rC}hbyo98zJWNVP3?v1SlmkQL!ICoK5J@;h9uAQN{doxh z+FY$|;QDCQKXm~;gn%8HOoD^Ko}Qi(p3)M|uC`!Fd3kv-L<%e=B@R%CyWJy_v0mat zx66M?K;zwTuJ$Cdy)zMXED>wv>`q1q0G9qS1p?`Bu|&5&-2^xc?1d$PB_$xoQ~HY# zgZa;-1j66cZe)G@zwG_Dg53=7k?>%ByqmMTD-KvVo6EO?m(mWacvqY(mt90_}SYq+$ul$4y5 zs=S}v91sK&pc##K zwfDeVtGYT9K)+8MZvW4=K>tzRpS;%pYzyokdBK1&;N#`}f6M*X5D-JhuYU(G(D^(3 z@kAizU4h{4YIq(&Ma6WWj#e`C8d^&9(Y@8T-%z+@1zmEy8wV}FY^)7^{;pguEw;E- z5~G$(UUXH)m@8dO;c^(|PHNxKKj9R*%dj<*#CB{Q}{h?6HnW&!PN) zxoSQ6Ga=rukcoa<_+|Hd{Tqs_CR~fJ6z=6kfDBKx>#@;J#(;=)K_yonnF7rpJz(hf z9?$^a{wD>|H;^N}s<2jj`nYp-z-y5X6bs(U+4~~8gSe`l0y|Y|A zw$PvDh+G`V87py_itq30g$N1RF7{{bZ!N!#VlyKtWg2A(2nf{H)yZ!W4T;2+BEs<2 z%7`O|o`In-Avh?=fxdX**fR?n;A_{Ko0?*g0f!yPqt<6fyK~7x+KF~!m56gs!o$ng zYU!z^Z2a~%(se8yOE8?flgrFuxMmMV$H({&g8;2DX5iv1zMt65NyAy1h zcYZA3DCwn~Loe;a5szU4o273q68xs%H7ax+9%?#Z8Q`a{uirLd0do}w**M$SlP+sE&5HH9E@zUFX{}Xt$pinC-H@ub1ZWR^dbsE2tO}cXNNf3Kx8}xd8_p(@ zUGmN8Jwary@2=})p=mLqWXNbQ9`n~^G#Zkx5hR~~h`kGVm4}5zg?oN57f038oPYAtV%ch?o5=}eZEY=t zs)2EqA3mKn0xWA{!aT#C+pnD;FD<;^bI?KQk9HSLKP~)d>gSP*43G&DM!g3mA>uGKLXND2xH!eE0w-&)mX5NCz# zQ)JLkIDAhmM+xH3!NHMJxhHZ>2IZBOdrnQKW73%J#PjFRY3PvypxO2$O`Yeza8tTU zn}L~f(=RRk?{&OFG;c`uj;3ar6fU$u$H*c#Cf;$D%)44v;9Fwk-DXR^N!zr>ni5MN z7>%m3lCBxN1+=FIaP;4cfRG-@+=MJQaeO3B$}R1UxcLyho3b-$jbigQsetF zaqlBdjbS+txnLG0+uQy%5$-Nhr}kF3r=#;i(@$}lfbr`4KrrqDn?&z?Jcgyk9P;i* z+ROUNTwG(3L!XH5e5A^o2lQ-jgmvq$KUx#uY6=OVnckWR<@2!oX=KL7k-6RG@z98? zxIFnMtWL+XX=6lYdof1b)F#tv=CmR`$t(z^^(o7=`0CZGpU>20*NSgicU7v35 z6$0&FasOb2d~V_aq)^9!?BQbPWFQ|3CHpF;!_03uVp$kKA`^Y_ass<-o zODo#D7@>Z<&004#G#q-;Fsq{#)Rlo|Ed7H+Cr1O04(5B(4Fb{^W3|mvmlKsHG6H!_ zPijMyCepOx^!09|$}kCXcsyQG^dyY?R^e^TF!A$z8W*tyJ67p;?bS<`mtRrnFh*KXjvnR(pU!(T7S3@mUKDDeV9BlR&M#b`07xN+gDWk-L@fya4 zQ5X4Ng1m7ksp4sVq({{kw1Kx1Qx}Dw)m(!PI_h;fEjp(A#%R0m)s-nfaf_b0Cv4z| zurRCkeTYzbQa5eADkf~b=gGiEfE1Pu0xh7KqfMp^X&x}0wvJFBFaOdrte%iFfpOQ{|>cETk*NW!8 z-7e}>)&0H!Nb-$KD#LmC`8?ijv69r5rh4mk7C`~;`BhKF>l2Vq!z?KRo6l6AzZ>i~ zwz09XvlE^|XHQ!s%`qFfOgBBBVbwP?3-_t!zw;5jMtAFA?`K^|l>RMZppaH<^&(}2 zq3*8uR?KsfH$}k9+Z#^J%@mETDZZ0$qisYh&P{xcFk5fZIWjn%S%l|r;M%w`F)_h= zc#z+r=q4t2F?f_$(YM4CqpYm_C>RocuDbPbqAt|MP&fSz`MQ{x(@=rE*BEE6hsN+) z@YaP#SuJk|brgP}f%R5DObW?(g9Xu95ho6E@kP7Av){G_Q&Uq{oPLGJ2o4oGLLGWD zTn)5oW!z@9E(WOGA^a4DRO}uJeZI&s(p(i8?~@aAkD}vCsy$hOrpd5t7$S!~ed?|e zSOV192;2}QB@?I>Nr{QvH>`GB#5W&X^EN9vO>oqr=4v4)nYzwI{#tJVNB8-J>M@V+ z?CdB7pT=8TciMR0C#KR!i~#5ZC|q#}ACI@HPeszXGafLahEkM*V`JGPU!2k(>e1@| zz_GFi8B;vm!P&NV-i9%al@%MGRC|5z8SPZ=vkB6ZK-}!>w2v*OR=&%H1C=|IPfndW zl~b`b^x1uJ>8-}_{n^%_Q}in(QxTH5hP!OvBY>iDQPp#W()=sGp%YTVe(6$3IU_f$ z^C*(gnzVD5E+&R!iKy)F@8342tYj%!c)mD8Zq#M-dl&DuPo}hlJ7d$&hGTynBLeu- zuPTohnl8SFx`zD-fAZuqXVi;9)@O9g4{c_?z8FNo(7ow}v+PdMp@tEyxjx$~eMO$C z!Y*oBB9<#o+4eAxrPxLZ2HU3l+Ve-0$ZBz4+nj+P z(X#4QZ);_>q>F0Cv$2c}Vf`OB2Y3A&C01vCdJJfjHZ{77EJS#V}N3CIzi!}Jdu`)`Yy zbY<{U@-*Psh^qyJVL5*Onmgf$q$azuK*O$frNGh$0p4Zjyxs$4XX0BkjDk@SBv$6L@ac}6@*#@wdob#wJrNqTL<%I<)PnO81&3Ad#VthsmK%Fgp;$Lezf7`#N zl$6(Zv@mT1)DkWz(RXcJ$pF02AgJWxxCFKE&r*8!Lq$!kBLT{2^=)W=ye63M-t=>x z>mFXyJmfP*tqI1oymA4ZDM|{ba&^V_=o?i#DTR)=XC6IB1tqWNj%vm%#HbD{WZWz` zOHVI4DXenKwqEej_V=#3F1-&=U+QRTX6Bym0u!Hd$~pl1ZMvXH2CvCzTI9rG{t!sG zyT&4dzf#lEwiXAR7kb|h`d;OmYCB+;q8y0kluz0<$7NgS4+S$S(i&9c=ojzocx_jP z9w6RsKtVF~JRkMaPSLZ%#^_-nHFYg5!OPh;^HvL=zp&WbdI#*yrz`R{H#fiTvlZH1 zo0~{UNdY!L?t>hW5*rnzud7Qs9BXQ6L50Q3IO z=Tr3e$Bm5i^c!=U_h9kK$w=4frb@qkX49PVb&cMcrstOhSpi(1XKw7--Xue62W9g< zz6D_WM;9(H<+$uTR=(`xMP(Je%6L8VBqm;lL(YjkvFNZsha7uzsxdP1@ne@tctPNe zvxPu%{qlsZCwH}C`_-~oA19GxVIU_G+vp^Tyc?VKL>U~<@DwOhL5-o3-{l!B3hXXW zx9-%4m|Dud|2|fQ#8KX;&>y`GPt_hDkz&UXF8AYdsjc`;yp-m&(zTNm#MlV;RRQn= z(H~+H+v7UqKf8t|p4}exlbehvg$;qaM?#TcqsNuqD>}}3dhlZc^5IVC(=exZ!@)B%mu~LL| z*#(Y&mZ~rJ!v~}Lyh7SV{`{q_y0I#|F4el m|M-{SzgAfLehqozG*tr1F7Zu%+S>6iLUk2wbhh%{p#KAYLAqN2 literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/buffer/buffer_logo.png b/content/ko/case-studies/buffer/buffer_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..1b4b3b7e525d0a796046cdacfaec27aab25cc3e0 GIT binary patch literal 4791 zcmdT|`9Bl>AE(4}6duInjtob58vEX670i!_>fb@=t-`mJ%IUhOBzFq_A-#g7z% z+$i=lSo1vEq`%O$$BQKrxRPrN>1X_qQGvMI06A$D$=4|)oG)aahkWt>Z{nxcCWz*VwwVB|G^`lw( zwC`GJaqMNM@L{Nt9T)o(>%pOl`whOMmZmW=K>IHSrkL#70k9&wD0tL>f<4Z}9*4yW zx&l~4{ox2~zjgzc$ znUEDCi{1i3v+o-|+`DlJ$c9yi%R|o*K(_YoMMHHSw!h#lVTYV_-#e}1xg6<;*&emF zsN)5Sv2s_C=Hr2o$I6fuy4oavD5X#mDVc5odf-0{(|MDNDTsT-lZkR~)FF3fZ#MrL zK+>gE+$sqncpxR0CcQsxlIaE+z9LOLIifzlKiW0{tVtHPn}S% zpCFU5Ox<;O$WeB63&?tW#>i!UKyOP+HK2`=1rZy+U1@XP5ejvCvm@vUB19bixZ1o; zX7i%nGG$KQ|0r}8-p)13c8QmtdcZOgf!9#H z(JX+}9ltx6c&HM+wNr41Wk?+Fn)~D|M6=1E%KpQkgTGIzBJMr?-eTM>A}|Pr+<}{? z2EKDQgs<~P7fMo|Jl)~Si3(AJ@GrNI?6hY=isgl{`&+ZK<+@T$XTR6)!trqn7<5+P zlzWH0N5#X^b(IHA6Q8fsn$|0QH|xfe4$7yqrThKA-L;*1Irz=OP;}th%Im;_$$;?( z9goT?T=%2$<4XN{Qa7Wl?`r>(75RJhd#ACW8N2?$1ZvF0AzLGYe$1O>56JIH*X)YT zGZ6}+!8BRy;9AVJTa{)ZPNAzZVJd4W;J3}Ey>+@l9UNCbwkEt$dM@(#CVip!RI~gz zBJy~t<0*?6Chs_DrFDPoeurCLgey{Y^PcM3TBFs^U@dYrvO_5+DbomjZzwCe?rr|h zy-cUb#(;a^u?aw#h(jhP7&G*ydb9q!{rUNrRp*-8YISX2x~z#PaYmEjhsSGWRekZG zQBn(i_pz^vMl3zKz1>jNxx2@wGhtgqw=cCD4(EHt2A4tsPt)|CXzDDVH2WzI-i(&c z?Zr~Uiq&X8yc&LxxlKj5bHjJdJ{j!8^o{uB2J!7(5|%-*fEy1qm=_~gv6}CgXbl~l zdbrv_gYP}<5o@BKb?!KDLsIYROiE8jo@`R5v4eGHw{cdbs9`gVM19@3<9Au`*E!|7 zQ7bQwCC2%7hHA*d;!BJAFv)t1zV$W};Y4>i&Dy+X10z=R?BUl>t{DZ`g|pE%z6Yv4 zh9@JT<;!+blWt_dCY=RjaR_nPAt4|(*xc|x|Ll{#bvBV+KY71d<4gKz$QhlHm8ITm zWmTV>LPV9uQ*W!d&qxSAvtI2S^7kzlE+|~k5VuagRfuxtS0&AzW7EXI zbK^LA2U_iPnL*J<#NINwkldc^kDOxR+wH#Oi*ksgtoZZ;(8mQt?u$YE6$90PST$Pf zpGY>#GI^FmXNMRGbp=>#v7=TZNlL4Y&a!ScVhNNft!Cb6i03_lmjlwAC7pEKeG@8} z8A|a7)Z4idyROS3E?EAFlypG#-P8 zSs&tSM$Hhy*f{(xbD&S7?_;B*x|5{)5XYLTQ9+9$6imaxSuX@}uEd2oIyOeHgZzYvuTY!Qm z-TDQ)bbUK25%po=Hdq^LTfSB-;_uz9`CK$`0`rh+aEHWO-?b1tb8R`RV~WMN%>ddn zBA4;H2tvJkfsBH3wQwZ_5+6}t-FaF#Xu`50I6u{`gnJwe1`2lm;?nUl2;QEnho`K2 zHm$zX`P^%wR3lP8f$TW4bWKvsM7-Ay;J;j@lfnqHh8mSJC`9`Ka}uSQ5_g^6STg^& zzAC_GxuYdAzC61>?qs+@k7gO%!o0eM77qS5Klrj|{Q?6dG{BZqW>K8F{aW7;y24$H zk}B`?gAe#dW&!fr=<9N2ifTMdb0Jx$a>uEG%ad+-`qOkmDYE$Ue2945r4<*2%+FC* zS(?+O#rV3kXmV1VcTVBbDpLeAzcKCIwL73v55&zPOEbh(1tvx460gCK(b#<-{oeOT z(XyxODW&9*3i|OKgtCsq-uO$Lz0-UtrON8j%_rn@cD!;)k48e}S2N`AZGImIvkVGS ziE2#ft?Oob4mh_#5JyM+-1J8gZg%v{F;un`@6OVEiMf;q$_OxMQR!Z&|L$G8&SxZC zJDtc?J4}vlif0?drFVR){=!>>eEJiBXIG35kU9yX3(8LKd&Vlukr2=w+O_Yk#VQ)L z{+Rvczc`-euzO`ox(<0E`2s@e=Pz}I>X%0X$7rzkjW(JuBO(ccyoaSb`TBrAWr=qH zb=E$8b@<@jkyGwY7$2YUo{6=sA7jSre|ZXj)OgdAu_U`II2g$C7@uf+bK4}iNEe%o z0Vos2zq@98c|ua-JI{{<2g8JLX3*t9uGv1#L5C*n=A2`Q&GkzfZ$$DSqH3y|mCKr>S$Z2pCKQv6JEHnq~u%Hu^2LT z1b024#JS2gB+-CDN6y1<3hZlIVr>ZF&+q-po;8ghqsStKbP-FML7qNpLnx zZj|N<4sfx3gwbC!s;)oj$r0gl(vW+%jGC;S4@Vkr`@LGDK=E;ye+9%EuNd7RiEeu- z8Z_;f@x%B=uohT4(Kr~to$(;hE?XRVTHm8kMgZnr<;9gxLeDKBr-5-f^0&+4lXkn5 zoLIfS#NIsXH4&y%SMjgCL(W6AU@uE?R=8*-657+`2>)jLT%n6#*^BKf_G{4d;s5QA z$4u6Q?zXdte+<9_TwG{HJRxx@HWC@i+Crm@mdswJCRqW`{`Fj~IqkPMzoK!oANr#U zR_rFog|(1YItq{k#w`B>v}p}Hq)}gaRMs$Ez=8>7m_pBz3;rev_TamFGs=<51ruRpl;h1xXzXbEt8z5xR zJ|F6Q4(_D7hk^UDQOL~!_-8(_NYzlWp~9^PXnl3D)p*59wT>4oyql*qiF_-~P*rx> zLPz2vAM*j7Hk3EzFO`O^)%;UhMmoy50MX9aA5#D6>@Nlu8RJA&)3~-ensj-D33VUT zk|7{<)4*k-+pEiq4q(jdOQ5gqtYOQ&S=E&fS)k2@X?9+1pD@kyZF>y79m$4gq)k)^ zUpGim2%5}jcDRsUn=g4Nq5LNn~ahaL!?7>>Pju<1xXdsTbbh1n*s6 zP@O>+X}Vl=S36rqfR)&uy7NVTM|yV1@JNMtur-PcA135moaSOlKDk-k!1bB!o2&Z> z));lOaiB>FOQl)yVr2O8%x}0UJR4nm=+7YJH~!UHOP_KRia^@=7x2pj;R&v6f6L^!t`4Y8eZVp_K)ZSaaolu;;7JDM!N=Me@iZ-9sk(EV>#PvQ+K4>qa{n29_ABHQ`M?4XsVvg zd*97woH?-Efb!zn!>1m5x%~jIXlkeNaIX6?=6;V@!_OR$NP^!|1N_K~VDnbKlRy8% z4!B$q)eIv+E-d6!51b~J2)x}sP^YNT;#!He9waxgE1oMpWfMefYZji56BV1U{gCXH zs6AG_W7(KKg{vVLSC0aK`LJZZDoVtJjbq5eu$EZyR(T$qnX}*)C5Kv`*V$Lk`oCw! z?B}5IA-5SUgnNyMBN{GRA>N-8ObFsq57YJBkNA>IgY9+?wBuuw8s9(NQ!bk^qw*xF zpOI6SVjSyYyImtWj6kQPOIsbRPY#B^NNaBAq8q+QyTqt7B&al%^ncrlzxC)vZ+lkk zHooQxNaNk)p+-r>JAq7$5md@6?X%WvCnzxEjg~jl9N-NJs`Y( z4saZtkUzESXRx3!gI$;o|M<`3uj-PxKa95?+1;z$9sT +

CASE STUDY:
+
Making Deployments Easy for a Small, Distributed Team
+

+ + +
+ Company Buffer     Location  Around the World     Industry  Social Media Technology +
+ +
+ +
+
+
+ +

Challenge

+ With a small but fully distributed team of 80 working across almost a dozen time zones, Buffer—which offers social media management to agencies and marketers—was looking to solve its "classic monolithic code base problem," says Architect Dan Farrelly. "We wanted to have the kind of liquid infrastructure where a developer could create an app and deploy it and scale it horizontally as necessary." +
+ +
+

Solution

+ Embracing containerization, Buffer moved its infrastructure from Amazon Web Services’ Elastic Beanstalk to Docker on AWS, orchestrated with Kubernetes. +
+
+

Impact

+ The new system "leveled up our ability with deployment and rolling out new changes," says Farrelly. "Building something on your computer and knowing that it’s going to work has shortened things up a lot. Our feedback cycles are a lot faster now too." +
+
+
+ +
+
+ "It’s amazing that we can use the Kubernetes solution off the shelf with our team. And it just keeps getting better. Before we even know that we need something, it’s there in the next release or it’s coming in the next few months."

- DAN FARRELLY, BUFFER ARCHITECT +
+
+ +
+
+

Dan Farrelly uses a carpentry analogy to explain the problem his company, Buffer, began having as its team of developers grew over the past few years.

+ + "If you’re building a table by yourself, it’s fine," the company’s architect says. "If you bring in a second person to work on the table, maybe that person can start sanding the legs while you’re sanding the top. But when you bring a third or fourth person in, someone should probably work on a different table." Needing to work on more and more different tables led Buffer on a path toward microservices and containerization made possible by Kubernetes.

+ Since around 2012, Buffer had already been using Elastic Beanstalk, the orchestration service for deploying infrastructure offered by Amazon Web Services. "We were deploying a single monolithic PHP application, and it was the same application across five or six environments," says Farrelly. "We were very much a product-driven company. It was all about shipping new features quickly and getting things out the door, and if something was not broken, we didn’t spend too much time on it. If things were getting a little bit slow, we’d maybe use a faster server or just scale up one instance, and it would be good enough. We’d move on."

+ But things came to a head in 2016. With the growing number of committers on staff, Farrelly and Buffer’s then-CTO, Sunil Sadasivan, decided it was time to re-architect and rethink their infrastructure. "It was a classic monolithic code base problem," says Farrelly.

Some of the company’s team was already successfully using Docker in their development environment, but the only application running on Docker in production was a marketing website that didn’t see real user traffic. They wanted to go further with Docker, and the next step was looking at options for orchestration. +
+
+ +
+
+ And all the things Kubernetes did well suited Buffer’s needs. "We wanted to have the kind of liquid infrastructure where a developer could create an app and deploy it and scale it horizontally as necessary," says Farrelly. "We quickly used some scripts to set up a couple of test clusters, we built some small proof-of-concept applications in containers, and we deployed things within an hour. We had very little experience in running containers in production. It was amazing how quickly we could get a handle on it [Kubernetes]." +
+
+ +
+
+ First they considered Mesosphere, DC/OS and Amazon Elastic Container Service (which their data systems team was already using for some data pipeline jobs). While they were impressed by these offerings, they ultimately went with Kubernetes. "We run on AWS still, so spinning up, creating services and creating load balancers on demand for us without having to configure them manually was a great way for our team to get into this," says Farrelly. "We didn’t need to figure out how to configure this or that, especially coming from a former Elastic Beanstalk environment that gave us an automatically-configured load balancer. I really liked Kubernetes’ controls of the command line. It just took care of ports. It was a lot more flexible. Kubernetes was designed for doing what it does, so it does it very well."

+ And all the things Kubernetes did well suited Buffer’s needs. "We wanted to have the kind of liquid infrastructure where a developer could create an app and deploy it and scale it horizontally as necessary," says Farrelly. "We quickly used some scripts to set up a couple of test clusters, we built some small proof-of-concept applications in containers, and we deployed things within an hour. We had very little experience in running containers in production. It was amazing how quickly we could get a handle on it [Kubernetes]."

+ Above all, it provided a powerful solution for one of the company’s most distinguishing characteristics: their remote team that’s spread across a dozen different time zones. "The people with deep knowledge of our infrastructure live in time zones different from our peak traffic time zones, and most of our product engineers live in other places," says Farrelly. "So we really wanted something where anybody could get a grasp of the system early on and utilize it, and not have to worry that the deploy engineer is asleep. Otherwise people would sit around for 12 to 24 hours for something. It’s been really cool to see people moving much faster." +

+ With a relatively small engineering team—just 25 people, and only a handful working on infrastructure, with the majority front-end developers—Buffer needed "something robust for them to deploy whatever they wanted," says Farrelly. Before, "it was only a couple of people who knew how to set up everything in the old way. With this system, it was easy to review documentation and get something out extremely quickly. It lowers the bar for us to get everything in production. We don't have the big team to build all these tools or manage the infrastructure like other larger companies might." +
+
+ +
+
+ "In our old way of working, the feedback loop was a lot longer, and it was delicate because if you deployed something, the risk was high to potentially break something else," Farrelly says. "With the kind of deploys that we built around Kubernetes, we were able to detect bugs and fix them, and get them deployed super fast. The second someone is fixing [a bug], it’s out the door." +
+
+ +
+
+ To help with this, Buffer developers wrote a deploy bot that wraps the Kubernetes deploy process and can be used by every team. "Before, our data analysts would update, say, a Python analysis script and have to wait for the lead on that team to click the button and deploy it," Farrelly explains. "Now our data analysts can make a change, enter a Slack command, ‘/deploy,’ and it goes out instantly. They don’t need to wait on these slow turnaround times. They don’t even know where it’s running; it doesn’t matter." +

+ One of the first applications the team built from scratch using Kubernetes was a new image resizing service. As a social media management tool that allows marketing teams to collaborate on posts and send updates across multiple social media profiles and networks, Buffer has to be able to resize photographs as needed to meet the varying limitations of size and format posed by different social networks. "We always had these hacked together solutions," says Farrelly. +

+ To create this new service, one of the senior product engineers was assigned to learn Docker and Kubernetes, then build the service, test it, deploy it and monitor it—which he was able to do relatively quickly. "In our old way of working, the feedback loop was a lot longer, and it was delicate because if you deployed something, the risk was high to potentially break something else," Farrelly says. "With the kind of deploys that we built around Kubernetes, we were able to detect bugs and fix them, and get them deployed super fast. The second someone is fixing [a bug], it’s out the door." +

+ Plus, unlike with their old system, they could scale things horizontally with one command. "As we rolled it out," Farrelly says, "we could anticipate and just click a button. This allowed us to deal with the demand that our users were placing on the system and easily scale it to handle it." +

+ Another thing they weren’t able to do before was a canary deploy. This new capability "made us so much more confident in deploying big changes," says Farrelly. "Before, it took a lot of testing, which is still good, but it was also a lot of ‘fingers crossed.’ And this is something that gets run 800,000 times a day, the core of our business. If it doesn’t work, our business doesn’t work. In a Kubernetes world, I can do a canary deploy to test it for 1 percent and I can shut it down very quickly if it isn’t working. This has leveled up our ability to deploy and roll out new changes quickly while reducing risk." + +
+
+ +
+
+ "If you want to run containers in production, with nearly the power that Google uses internally, this [Kubernetes] is a great way to do that," Farrelly says. "We’re a relatively small team that’s actually running Kubernetes, and we’ve never run anything like it before. So it’s more approachable than you might think. That’s the one big thing that I tell people who are experimenting with it. Pick a couple of things, roll it out, kick the tires on this for a couple of months and see how much it can handle. You start learning a lot this way." +
+
+ +
+
+ By October 2016, 54 percent of Buffer’s traffic was going through their Kubernetes cluster. "There’s a lot of our legacy functionality that still runs alright, and those parts might move to Kubernetes or stay in our old setup forever," says Farrelly. But the company made the commitment at that time that going forward, "all new development, all new features, will be running on Kubernetes." +

+ The plan for 2017 is to move all the legacy applications to a new Kubernetes cluster, and run everything they’ve pulled out of their old infrastructure, plus the new services they’re developing in Kubernetes, on another cluster. "I want to bring all the benefits that we’ve seen on our early services to everyone on the team," says Farrelly. +

+

For Buffer’s engineers, it’s an exciting process. "Every time we’re deploying a new service, we need to figure out: OK, what’s the architecture? How do these services communicate? What’s the best way to build this service?" Farrelly says. "And then we use the different features that Kubernetes has to glue all the pieces together. It’s enabling us to experiment as we’re learning how to design a service-oriented architecture. Before, we just wouldn’t have been able to do it. This is actually giving us a blank white board so we can do whatever we want on it." +

+ Part of that blank slate is the flexibility that Kubernetes offers should the time come when Buffer may want or need to change its cloud. "It’s cloud agnostic so maybe one day we could switch to Google or somewhere else," Farrelly says. "We’re very deep in Amazon but it’s nice to know we could move away if we need to." +

+ At this point, the team at Buffer can’t imagine running their infrastructure any other way—and they’re happy to spread the word. "If you want to run containers in production, with nearly the power that Google uses internally, this [Kubernetes] is a great way to do that," Farrelly says. "We’re a relatively small team that’s actually running Kubernetes, and we’ve never run anything like it before. So it’s more approachable than you might think. That’s the one big thing that I tell people who are experimenting with it. Pick a couple of things, roll it out, kick the tires on this for a couple of months and see how much it can handle. You start learning a lot this way." +

+
+
diff --git a/content/ko/case-studies/capital-one/capitalone_featured_logo.png b/content/ko/case-studies/capital-one/capitalone_featured_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..f57c7697e36fd06fbd418787ea15eea218bc7692 GIT binary patch literal 11306 zcmcI~WmH`;*Cy_6rMQ<;>;e~emm(K;cXuuh7bxx&T3m~}yL<7XE$;4aLtmTu@y_>a z=B#y2lAZl*%g)M4R>GAOq|i}_P@tfo&}F2>mEX#Rx7QyD=IslklXmo0ymyt*bX9RM zcl9uGHiHr|buc!gkg+qeFjF=&GWB#CHsgnaf_1i1)pXUA|IBORV8?9qmxmc_=lF(( zg5nnfI~tkTn7L9In^{=d3s9Z5c2ZGTnF>&80OeWb9mUKnt)#u2%~ZS;R872WOn6MG zgaj%0!Mtw*?95z^D8P2M_Ab0&0jht<<$bIFC9_ab{DZ~SMu6&{Mrq0`QHVJ>n^6Fn zIhjlV9IO;vJj?)2Rt_M5k%EmCz{SD}WMSoGVg>NBa`UnQDE{?Oy=ikcHRn|pm;9Hm zx0V2vrK_tWFAIx@hX=C5YTQ#nax^2+U;fLj7+E;$|); z&Q^}DRu1+Qemh)*xuz|fqDxW3)sk!1;EV8VrTc)uYYj6 zxGJ0dZ!`WYw~MN$qZx~`nTvy)v&q|dm{b27{1&_aSMy%XyZvkXwGh;J01{j<1{9Dd{ z#RC9hJR(3K7biPA03gB5!zIEc&Lzq!$}Yym#SWDGH&({p#ns5(#O&X?t^Vr%Czj)X z#PW(cn;E$}IIB81*#0{Ulq?-w9b7CO94W-a{*F438Aw4dZ)9R+|Cjvn?@;~Ib8$0g zD|a(fNoNN;ihsnK*XqC6!Op`i4rCJ*XJ?ZH{`G-df{Rm3lwAbKF2>C+A@b(KKe49& zC4T?HO8j53EN{lJ{7sbqBXRysddtSY)&ER`x5j@al9~P6>~VgZCgxCp4^U9_3Nqp% zs^Fy~Jw!u1wFiF{cMI=?iJ9;?TMOFm0<1wVb1y zmSIf^XH5kHBuB&HY$aTS`#9HK%a}7Wm8^N>w1Ld_k#N`foRhfNh#u!95A>O1dS5&O zWi02TuxdVOuo1_(F)RJu)6k}fJ=f_u663B1x6jwFFAoR}#UCweQ53kGo8eN@&K{7^ zgMujx6Y-ECG;oz4EM1JFu&EHrq?NpwYWgVjDb^iQvF`(fXY%_+`v;2?647~@e<(D6 zcIB|8G#;x{_joy%p~p9w5c*14`3{O&Np*7F^!O&bH-@3~{=j;C|4OI)C6!1+4PL{c zz{hW70tZHlA|6c(6o8e@ffCP=D;-aTEi#e?FPz=yKneGznevHgBj`-34VszR&=3WN z@=QDqkrN*x0$PLWhY^JlsRjJ_w&H7i>0R=8HGO?p7#T^;K?EO`BG_16rm2R%1tSuy z9T6e0y)CK+57HZ(`&ENJ*7m!%)(^4^WsV0GB}F4L9hIsyr5UIUCEP_qYq$lC0r&=G znJ^x$L@+>@@Y;h#ozIke6&v*@jy<5W&miuu@haNKRod5NW2=}#5pE%1g8a;d1cgG@ zPFij@=6%LhD4&}iFRmjo3Koks^d?$Pz-T}4cUTK=M^3kj#_1oOy*~!@>T|3RdaxGd z-auanJ*Zq1E$AM#N#*F&4kqKcqrJ+9JUZ)&l|D=PVi-gv-Og&UJj9vMGu7;g>ah*gNRq?pOWFNlSFr zsM|ZhX?&jX{<@)B$m1En!LxbK9pnS0N-~*}I;lj|l^Ic}rWyF9Sq{sS#;}2=SAaPW z<5XByD;o}-wSSEQz;6}>iKRRieql(bS(8)JFj#WJ!_!bnQb3Ns%ZfJ%?R zThlN22|I1?;hk~NX5Vb!Upz4k3IKP6TzUn`kKu8n)B|r|ANBZd!X7b8G?Pm!QFEcL zsP}Y5oF8a9V-tSlBjwC~qn6Iba-1YS>b285pdR2>EHAFk;_tu{V*(`l)rg zW|5Fe%afWC8o;l=)eEZa! zm$ZB)Av;n;q*WJA0;>)sis`3&%`iUa=^1`kJd&}$78(LlyM8|X$}VEc`*gxNYjn*t zB|Ro1JL7Uriz(>ab)24Ub;@|jV09audR~3`@T7>Y7TsKKng&W;h-FYeE1XzZVzh#? z*TCcet4eNM2nB${eN<-4(Nm{PV07eL9S3}vg;cO`fROq)4@tv@@#92V7(_4dfJkw& zS4Qtg4q-wLvP*h{H>Ox4zppz6?z=)xBDlQFFhFT;F4EVCX-nw`nBXq)%w1ik?j^6B zmH{p0r7GhPB~@`wK4}`I0ea;^Ts769CVmhwbaJtzQKvZ(i4rFS#K5zRO-n|nlu2_x z;@n%_xW%kh0(H(cAP*snJspbgs>o;%2yshmY#oj0KKnvmE?Fw4koS1?0zJVMpV0@0;*c7GB0vp`)q7haHQ9H0RXR@cC6o)Kxjj88>2*sk1`I$HR%) z`tsF_jg9yGG?bw(%(2jUzQ^agBb)KEo}=rDh)bACU&pA)vT;Q%dKf$QS5slQ5Y(g!#)gu zPtU2MsXfPHI)TIana^VJgVI$szGZ8b>v-bXE&T2SQBn7o_{7y}K94tia&o|S@At~e zmS046*&h6h8Ej%cwcDk8JoD1E!w!JB-{<pi>rg|mm4#hX*OtWG$-8! zXsfXTbdVu&qZZ0R9G-?2b`;z7N_!Bu1;9T9n}vE^FEd~@l3rH13EZK8CS|`hLFEJ~ zjp9-^%`O=!H1i_kmEL#Vb&R85cDYCL-io7lhNJ6OiuzGrK>uZ5=^@HV1|nHewDWo3 zQw+IUe_y&zgL=#E1@@ta0U+rIPzJmH+0mFIjYbAy`f2nIW7szaZ)OU#`>>M4y|GOt z5}L;fOa$~(FwNMfigC-Rqc~sYRyOFmU00C_`AD(tnw%%~gr!9OYQN{iJ~PIP$eC3a z4@T>ca|-*)u9TB?J(rwzl9Bqld2ksd`sU3$WX*DKW%jycIMLBMnJBwM3OQ6HY2~`# z?G-z}?k`o$e{yGe6a_mlqw+XE!r{xrdyyvKIx6TZhirbqlR+pcZKs;|?EO+yTERQv z0agguGlK0ljN63|9YoCX1OKsHv_8mj)rQKB|_Vfbvru z+a|C~`a*f?R;qQb?5~gRsmo=9ySv;NM0}J3U98qYKdwK!JGhIyQs(`ek zR+G_XVBld^9%Pf!=559p=y1{}yGp(W+yS4 zdM+tbig}vpb)_H`x(vyyntQ$J{d8xdaY#%jsJp&_yzcc)$I;0jRa{;=&PbSa z1j&VineVry#~=m7o%)@)^^Rf6l(0d+82HD-9DB?xPn3nbp`5|-qdIl=`tAwa#=`45 zqm5&!?MMj7Y}RK2k1$0Y_mwUu-}%lT63N)Hw(OWsJ5UajdwTgWG?=sazJf9`t_Dw0 zWa&Pq#o{p0>(amh8_}A6><(z0IC%Jrh}*cQtlaDg#mzik%y=laRCo60Cqf@yRT;eM zazBTAhycD{uxoU3mOCh*tWH*yiZ!U07XCEH_M+s-fRrY|SqqaQ@;;xHoArgEE}(X5 z0N;^U`k$T3dD!!W{UXA%qxh`aW=mp+-}YhfSVn9Nt2xge2KlmO?!O(PDZtLNM(IHJNa21@0w6PZ9C5-^) zS#AoMAM69L8g@Ey^4w!JUsN6c?U&o`HQw#uN#BI!(^{4&?-WFXqust`>?C z6fLMTh`#3)j%zlZE8))+yFC19UKy}9^krSVrz3kRxRe`E*s5msm9Nu zO}c{qI3lRi9?ewJ=CZR8OWyuW6=8oSDxbK#y&0g8FgEq!eI@m-&(+LBySS}t5fM6f z=Ik7#GV?ei!za07T!)T-u;DIF|5N2x+#r#y8+E@&7%-XL?H}*Dn-RXOP*|Lc=@=%z zh!=xVCw=`Er}0CB?kW+%Ebkc|2GD|4%zSZudTl(g-}TYzYqiE{?Lw3wb5&y*??4fv zG9GwkX@e{|%R9C0j@ZhoM=q;UzZ%s|8g2JdR?euq%ei)FzhgJ1)IzbXL+ogH#N$u4 z{Uuq`cKd0g+X%POdt)UmAKS_O^WS=8j)X=ui_7{fqX}&+uUiX*&EklNH*FL> zo(D(v0b!$?tGcS;XcsX0!*fs@YzL7H#LAnwB>J5ZH0-#>qLMT$;KhL6;~+$*N6nNJ zH?QZlu^vyuB+31{;pC#aCPMBw>rDtp>r>eb>=sGMZGeMZw#jF@XJp&?HCF?ka~|7K zwr)t{t8~^tZq=Y9X6EAx3P^&Q_;zRHICj<9>>+ADRNo(XpR+Gu;1|#TTLw){UB_OQ zaWHRC@b2YV%AQM#n23a_IC;;dKkT$}SzzgNy($IWhdOxPbF`cJ4&?WHp5O z+cMKA?uD(2r|IRFKP1*rf*7_jCl;-b!c4NjrbjLw8TR|%5m#PL5+~-N=aqQ}E|*b@ zs&hPbM2_qlGf?|COki7MRJVWhq2^})*x!hF_{3*Xv_DgmKd~U``q=z_!WWY8Q#q$y zNto<1FiVWgpVT?ktKWw#T`je{ysl$XhewnrK%6v{E= zbAUE#xd#SGk*S`aZjwxaC=miV5*-nL2r@qv z8RnabvvYXMH9s?CY97b)h|ub4WT_bjU{lZ$-e@HS4G^s);^wN_f4@Ej^G%!Un!Fix{}OdF(xclvv>wo%dymm3vQn z%5qS4GC!rBmSpc7QOS-yqS5EBFF0qJ{UhYbNNWG*d?!vlTK8mAG<>j)J5PHhBEzM>%VKyG|9B2glEow5 z8_qTWRP2d>Wg7EWO{TRn7p4WoJ=@rh?)wJ@vJUbLpzfD$d09$tG`~Oh_qnEkPsNS6M3&Qsq2GU%ufWA?=QO} z@cAM!o|+EF;A89HjpY+-H7r@OlE zrL0rGtXytzB%BXD6fAx-d-1P?y z`&7+kR;}NxW%dw#I~@zP)8kywP=4ecFq@u_ZbMx!aZ_lhZvSn*h*;f0h00BNH z6?t2=fpXOPjn-*cGQDWS$$A|9^9K4ORG3d3qJkJftu7JPE5=#1%n!?UP8^cEiH=^R z(A%N7j1NZiLYKj2wwL5B(GU3Zfe8Z>Rb-3wSi)^6IxWsT71@(qh}2!=YjP!JU17nt zAB|F^;8GRQ2DR47cKjMJ@OUq(n)Ued`!lVcfVMfQd0iCDg8H8MCFg5_);`3@I{?^L zi5s08mvwUUb^(`zF?;k)hXn}jlVCQmB^M)20S zw|?{edPvMGuXLLV1+70ARe?Ls73q|nfgT)4+i@Jfis#oP!AFMBSbdisT*1WOEQ6*oE2~HKvL}nfU;k(CjKTcn`~9ZfQAs7JbN4Tz*_; z{+egT>vgS~TL0pyEN0k5s>A&hz3TjH0dbb^x{JwfUD%HA#xp*ukb%q7hbqk|s18hx zQz#)@!_)Kz`lqAap2rifk1dMs_X*8*4!I_f_x*%M(rO$F7t|E71-E1J&*<3ZN1cgaoMJ`u zAxJ23P$b4l>PO5RD+eZ>4y#U!=hf}FIC9C>u-nc8fBh}~(p!>_(?BjIFWvoj6X_gy z?kQ>}c~bJkb)5p9=RX5112D3s@roZw={NzhufvUL25CMfbuX^$HbkT+{yv@e*AAS> zlgclKmQ!$O29_a)P~I68Z%e|(B51-G3+ts9%TWtY}=YHRK9*% z7A`;(Y7D-A`!eOF&Dr~-v$rEFxr#QH(A(L%Sum({On#X)@Mzv1gSbxKLH7 zT*yRhkiY;Cby5M3Nyh!cmef>=f{vX4_l!&3z2qk$s$t{ybVvSl{-p{MdqPFmI@`d{@G^FL4pV$Q;+%u@|W9xr<^p4{+^zPR76 zoqS(&!?8=E9z<1>XA;b+^ftd(F{w{1WD(z&Sxc4W)xh(8;ZF2BevEZf5a{mF?0nTE zZVHkc6-#l2*#*s6vQEk{T+tElNd?)9iGZxsOmx_{A0fGl9e0=!kydJ>{Nqp$y|e9b zB1M(v?(+GMCXWmpVf;FaEF1<-U4Qo-q{xZAi#fon}!unP?r#jul?YOnu7fsesdYHu0 zRP`B)9Sj&frr&Mkh$`*2#@fcRj4{he>c5Osw+jWH#yvP_$-uW0A4spqOGt&MVJ~XU z6c)90oowf^33JH>+`V=3&mpYa*R%%AnxgK-GI(e*=b@*CK)-p@9Bl}_1JiCy}YjHa(x~`xtIL`dsc>7GHz1SF^At2$+B0994R;kjrm}wN zvSWag>|=LuUczIZgVq>U=xKvt9twmgC&dS3IDx3-Cj0%|Pui)Ma$aEqyu zqWUOSKYWtNBt7HgtJY8m4&~T$&0`+6jYt<6n>51J;-0x}4jMbIur$w%p~2(Ax0xpm)i-64gLR!*GpB!Ufy!O^aH`vomk!9Gx%wjfoxiMkqFGFOtX z>S2UOCI5F(Tw|Ve1<@e-ex^hnfVmq+805wC`J*hu*YI$*eJ94^x)Nx14tbL?icU-v z=KT5b`=dcElhA{DqGZ&gkELE`<);Uj^Vk7;096_l=Id2<#0ep_?&m6!xg*L;tWbEu z$l+`&U_EYE3%si`$AuTapTc0eNQ!@z`G=Uw^M$RZ+m;TtsAPP+UWw4vU;g)hh6~AR z9v$(W-~OTUXchZ+Ztzc7#0U~2Q5jdNes5o&&{tU1HQvE)cq=NB3KbphPh_M&=ZB`# z$n}=Yznnb{9{MjaS-XL6q7p3B`s2u9#~JfzwMRqcTGEWzHXPAxALO5R#&j4CR0^OO z)=*Tn5qx%u$+QV`hY=7rp6L0a*$#q*S&u3s43n1s@M)NLmlxnKT!sIX!o{HsO*P6(K|sMeHn5mUFye z>DnEDgi}8Mc0ij$daKTVyQp~BfKSO8b}t!iU>3B8^g^RY_dXE(=Mi?8pv);Vi(^CD zZ25N89v{eCc30$B3{Alria1mzR*7-?bYF32>AT9O-vz5pTUdZpw5Yb(b%GQp&vXUI z#HKtqMIsD;F`Q`oCDZH0f3w5(>!`~yzi22;E-ppP)bT=Zz71efS0-+FX>oX)O~8WY z3$VS&Ch%8Cv)tqQjX~9lIN^(%lZ{zYf8Ask@B{`R9PePh@`9)3ihAHvaIM&MuhV<< z{qxY%g8-O+E-rqhWj;DTZ+K|rNK+7<4DMUxQ?VI9pe;P8d17&G%%NwOE|nEnK@~^o zfge`yDLDEsNc(+j6+J|TiC69VhtHsNzj)a!l~bDOIRteF8M|6J9jn?K26fXq%E7dl zXeySwYH%u{vJxR*+=$#G10-F?w>-=WZze`H?>C<7-8Do&{SCP_kc5&=c_!wpESJYw zuX5RtUK|b?>-Y6BG`t0a#6Zo`p1I1*?`=29B0{!I{HilYXLp_=Z|C!*IP$Q39wg~m zGqg;ZROi2wD>{FaMyLWKk`p6=jarv`VJwx}8m@t1WDajTuuv&6`N)l%L(i^{sP22?JqXsX8>@4HL5>$ zE&eK=34qo^^%I2p!|8Cow73}xCdqY zc2{!C{~=SwYlODyr{>`)=hgOJ6VKMG_4lofaBy;Qb$$LzN77?$GZ@9Iw12Rg1abM) zEC?osSj8G2-W<+HDgKzie&AC8R)U;l=A0{jpzr#|d2Cy_i*vXmQdPr!f9L0KKzC^{ zmD1Jv+V`Tj3nZNnJL#yppg4kxUM${gk);lfSfqC;l*lm8BUFqH4L6hDe(QOA3@-C> zdJON*VkpkdvV&w{YL;I4U4Ei|VE$osYwk?fuIi1~X7b8jd+KA|Qy*Vh;p;V(y#4zz zdawPy-l~wSX|0vX-`5YY*LvWvJ0LToF+cudI6C0lH5mDBoRBN~xd1hB*Pka>v5-$8 z{3ty>eay{u5_*c)O|*_uwo-fhFj{kbFWHbtrri;E3y#Yu*-m%`Dd@J9WU71vUBfpYry3TcfMt3+~o7D~$ zvS-vxtAqJe_1wkcAhhy{?|lcK{)%Jok;K?qU z@^y=E2f>#6l4UD@ljJD-xX{x}(sw_%%4xsmVpwG_fZQ^4_&D|ag%6~m?>Q}aTCu=+ zdCdZ12itC-Z-%ymC*0nj1G;*^)zY40N3s4(x4i_WzXW(qj*eO@*fZEDUw#z|jio(9 zv+ZL#_`qWD@9VSxk*aDkMP>*TVKwzKng$>E^bZNGW@IPzuih+G+gIOW{Ew% zW>bH?{Y-qhQg&{!ea<34F~`BvD(|Z{#!AEV0f}mE`!FG`&2z;SLcuIUWoymLZ&xlGXm041#SL z*!!(0sM&fuInhZH6B+AwmmFBg?1^1ne>?jwavq{_)m8qsTOq{#UQ_DSMDBlhxZRlq zPn^LmL9NZ@V>o|5Gw}2!sq|Gy7%r=8=O*YKj)-2syDMw$ybid-yRclh;c1gZxVmSsDpb0Tk&#~956$j)!YKn2nHdj-LQ^16ik%JpY!@>eCW`1->nZu` z%;&pwZYSlXOPB9kTcd6Acodlq=Xm&}B&?Hz4+#|=f^O^lzPtPde>*hRwQlb6VpCYW zR&AqBE@rGvUO!D0?CX6LZVN8h9|Y1i#y~LJQyAG}+xFS~Ju6}QP$_JEpI36kd@lR zU4yz=H9dMR)VUAp{G<}U729du?TUW(@EC|l9QutFK*;~HBb2I9HspEUUj+U972)k( z52X?z5=kc4MBHNa=|jD&?#^e4?;CvH!hMnbeJ;0DS|Ql@JnaB^kSz6+IjDy#f?ilv m92-9D|9h#ce&G*;2E~F9fcpCe`TXzgT^R`l@d{Bx$o~Sc%O{xt literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/capital-one/index.html b/content/ko/case-studies/capital-one/index.html new file mode 100644 index 0000000000..773db4869e --- /dev/null +++ b/content/ko/case-studies/capital-one/index.html @@ -0,0 +1,96 @@ +--- +title: Capital One Case Study +case_study_styles: true +cid: caseStudies +css: /css/style_case_studies.css +--- + +
+

CASE STUDY:
Supporting Fast Decisioning Applications with Kubernetes + +

+ +
+ +
+ Company  Capital One     Location  McLean, Virginia     Industry  Retail banking +
+ +
+
+
+
+

Challenge

+ The team set out to build a provisioning platform for Capital One applications deployed on AWS that use streaming, big-data decisioning, and machine learning. One of these applications handles millions of transactions a day; some deal with critical functions like fraud detection and credit decisioning. The key considerations: resilience and speed—as well as full rehydration of the cluster from base AMIs. + +
+

Solution

+ The decision to run Kubernetes "is very strategic for us," says John Swift, Senior Director Software Engineering. "We use Kubernetes as a substrate or an operating system, if you will. There’s a degree of affinity in our product development." +
+ +
+ +

Impact

+ "Kubernetes is a significant productivity multiplier," says Lead Software Engineer Keith Gasser, adding that to run the platform without Kubernetes would "easily see our costs triple, quadruple what they are now for the amount of pure AWS expense." Time to market has been improved as well: "Now, a team can come to us and we can have them up and running with a basic decisioning app in a fortnight, which before would have taken a whole quarter, if not longer." Deployments increased by several orders of magnitude. Plus, the rehydration/cluster-rebuild process, which took a significant part of a day to do manually, now takes a couple hours with Kubernetes automation and declarative configuration. + + +
+ +
+
+
+
+

+"With the scalability, the management, the coordination, Kubernetes really empowers us and gives us more time back than we had before." — Jamil Jadallah, Scrum Master +
+
+ +
+
+

+ As a top 10 U.S. retail bank, Capital One has applications that handle millions of transactions a day. Big-data decisioning—for fraud detection, credit approvals and beyond—is core to the business. To support the teams that build applications with those functions for the bank, the cloud team led by Senior Director Software Engineering John Swift embraced Kubernetes for its provisioning platform. "Kubernetes and its entire ecosystem are very strategic for us," says Swift. "We use Kubernetes as a substrate or an operating system, if you will. There’s a degree of affinity in our product development."

+ Almost two years ago, the team embarked on this journey by first working with Docker. Then came Kubernetes. "We wanted to put streaming services into Kubernetes as one feature of the workloads for fast decisioning, and to be able to do batch alongside it," says Lead Software Engineer Keith Gasser. "Once the data is streamed and batched, there are so many tool sets in Flink that we use for decisioning. We want to provide the tools in the same ecosystem, in a consistent way, rather than have a large custom snowflake ecosystem where every tool needs its own custom deployment. Kubernetes gives us the ability to bring all of these together, so the richness of the open source and even the license community dealing with big data can be corralled." + + + +
+
+
+
+ "We want to provide the tools in the same ecosystem, in a consistent way, rather than have a large custom snowflake ecosystem where every tool needs its own custom deployment. Kubernetes gives us the ability to bring all of these together, so the richness of the open source and even the license community dealing with big data can be corralled." + + +
+
+
+
+ In this first year, the impact has already been great. "Time to market is really huge for us," says Gasser. "Especially with fraud, you have to be very nimble in the way you respond to threats in the marketplace—being able to add and push new rules, detect new patterns of behavior, detect anomalies in account and transaction flows." With Kubernetes, "a team can come to us and we can have them up and running with a basic decisioning app in a fortnight, which before would have taken a whole quarter, if not longer. Kubernetes is a manifold productivity multiplier."

+ Teams now have the tools to be autonomous in their deployments, and as a result, deployments have increased by two orders of magnitude. "And that was with just seven dedicated resources, without needing a whole group sitting there watching everything," says Scrum Master Jamil Jadallah. "That’s a huge cost savings. With the scalability, the management, the coordination, Kubernetes really empowers us and gives us more time back than we had before." + +
+
+
+
+ With Kubernetes, "a team can come to us and we can have them up and running with a basic decisioning app in a fortnight, which before would have taken a whole quarter, if not longer. Kubernetes is a manifold productivity multiplier." +
+
+ +
+
+ Kubernetes has also been a great time-saver for Capital One’s required period "rehydration" of clusters from base AMIs. To minimize the attack vulnerability profile for applications in the cloud, "Our entire clusters get rebuilt from scratch periodically, with new fresh instances and virtual server images that are patched with the latest and greatest security patches," says Gasser. This process used to take the better part of a day, and personnel, to do manually. It’s now a quick Kubernetes job.

+ Savings extend to both capital and operating expenses. "It takes very little to get into Kubernetes because it’s all open source," Gasser points out. "We went the DIY route for building our cluster, and we definitely like the flexibility of being able to embrace the latest from the community immediately without waiting for a downstream company to do it. There’s capex related to those licenses that we don’t have to pay for. Moreover, there’s capex savings for us from some of the proprietary software that we get to sunset in our particular domain. So that goes onto our ledger in a positive way as well." (Some of those open source technologies include Prometheus, Fluentd, gRPC, Istio, CNI, and Envoy.) + +
+ +
+
+ "If we had to do all of this without Kubernetes, on underlying cloud services, I could easily see our costs triple, quadruple what they are now for the amount of pure AWS expense. That doesn’t account for personnel to deploy and maintain all the additional infrastructure." +
+
+ +
+ And on the opex side, Gasser says, the savings are high. "We run dozens of services, we have scores of pods, many daemon sets, and since we’re data-driven, we take advantage of EBS-backed volume claims for all of our stateful services. If we had to do all of this without Kubernetes, on underlying cloud services, I could easily see our costs triple, quadruple what they are now for the amount of pure AWS expense. That doesn’t account for personnel to deploy and maintain all the additional infrastructure."

+ The team is confident that the benefits will continue to multiply—without a steep learning curve for the engineers being exposed to the new technology. "As we onboard additional tenants in this ecosystem, I think the need for folks to understand Kubernetes may not necessarily go up. In fact, I think it goes down, and that’s good," says Gasser. "Because that really demonstrates the scalability of the technology. You start to reap the benefits, and they can concentrate on all the features they need to build for great decisioning in the business— fraud decisions, credit decisions—and not have to worry about, ‘Is my AWS server broken? Is my pod not running?’" +
+ +
diff --git a/content/ko/case-studies/ccp-games/ccp_logo.png b/content/ko/case-studies/ccp-games/ccp_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..cbf3d267ba8ddce67119711911d799ef87dc4c3d GIT binary patch literal 20368 zcmeI4c|6qJ+sB8nGMHiPDqA5UOV(Coi96YnoksSu zL?~NGwj#z@pHba4cX#)1p67Y}{+xNuayjSvT-W)&&vnjxzu(tuTr)J#IzY=o3jhEP z=xD1O<39)RuMlc-{QHVwJp}(j^YJsxQKO+Z4xQRd< z5cV(;UpEiDHvpiZ?CSx6y1-Ce_An>7yCTohvpOCwxPu~(xuiZ=-$M=N4A=Jef|>Xm zm_q$spt24;%1X2fzH;~gZZH&t%h%1-9VzFl$n!O>9Nyk*2JvuxbwRl(@~G@3bojp%d7M!w4>=GBjYf;0#YGTaPI$SjEC?(H5)%`~dk7=_+))r;VRt0& zw;+GTQHLR+UT_Z-9O2Hj7Z+lW@J1=}@a!e}@%VmRZXQ1pxg)=`!&3zLLOejCB4E(( zL=Mm&8V_$T*RMl!fP!GIFgKVx3W?W={#NJVj6flf&WL|W`J?+^6XWMZU;jt$-;USK z?YF6sC=DMxg6~Lwt3;amdB8x%FeJj;3kuWl!5<3m?~Wb?cl6!MWy7#!E#_}VX(9u7)%i4o98bv^brnlN59|1kPwp- z6DN#8;Q4C|{2FqApdkOND1^QQaU7s>jtDO|2uca=262LcJlvfWKtCG^JbzS{oEpLv z;e{U-Oi5e;^t)yP)z@m1({V?lAns6@j=B=QK?DwWkP{UZa|Am=WrU%!(qLf;8K{IX z1nMXw43ThvK%}8ySy4x+uMqY13FZF?t&V_t@2!~cp&kB@p$X*w2z}NIj^FJdt^_`N zqyN9A_)VC9q|$~X@ipp4P`mgF{<@!;z&!un`d5c5{Oi8t0r5h@_Ew!D&tJ>#$42&J zpW4%WUA=M;=-!T|1l?N{Fb4(D-&%hT>xc5Y4F3O}z;E#hsIOlC z9t4SSM4=&GFcl|!P5f`y`?tt{m;RbU0krom;0piyWPEG=`yBmMwf~$`g}u#94qp%W zu|dAhgQx=NA8kJ~fAhE4n^#xwul~|vVEh9ly7&HFO4$9Y)bc+|3A=xl{#2T|!%<3N z-?RPFOrYA^fxm4HCis&Dn39yFq`0I4=-1|76^3wMn5%_49KVKkZ=*wAZn9v5k*AkK*B{xgQ!iyMHCUC z0|^%)4Wc#)7g0on4kTQJG>Flk7{xNA zNVu2zOWfYEUcv?+s4C*+zFYV3&vj;M8rtwMig48{s@&WvcW>M=)YdlS2lBHpF|pA8 zOV`E|8US7q0`csk3|HM#;)Q!kyl_v67Y5PmNRI_L#;h{XmzI@#YXw9^*k{}~p8Ch6%*d?q6;tw>jad7AYQT4X~2g5^y>y+o>DJCZ-CN5vTY}(P% zd*)8M8rwk*S=E@wB_%xW@>Vk(96(R?<(V82vFY*7 zM0T;aQ0!~$XFlcBbt^R~UEtGASfSK|k!M*9}iWRIFUmdqc zkXr`J(Z-Y<9+?d^#}>*X_{z$TC)ht8tXe%<_3Rl?r;$xft`1;lJ6IdEiCt3W8su=n zQ>t(qef9eFAFW<%&5hT|R*T|hpBb#RYytMcB-h}W%zACT%Ne7=zP`ZE6MV9QS43Fc<}|_A zp4%TdVB;d;v{sm=v2vi-Atsw#B@6DU-*ln$q2o|#Ue@l=(}Gw_dij2Y^fr5~xltmt zCDt3QMG1k1skWDJ9ON`*nc`UM5jD8vV5t-|J^%9MjhS0dk<%jq*fvA3gC%Qr;;?-N z(mrmhjg4*lSy;^b7YoOZRTasz z483tF(Hb)GA#0Y^x)ForNKaFeL}bmlCMCs7x|}ORqe^XD7iJgAc(v<3f&>K8zz|g( z9Z_!J3WouwvIh0|BE8(Bj!x@f+16v22NlKGwxpfLM)HO?4|6~DJU@qaPrukQ@zgaw zh@SiX#PipF$Qj4!`~;v`+zDI7U$;`qBp|>U$Ltn57w; z0eTOdSMeoRA;X79{jG-S-hViF2K3erfVldr$0!32`lx&?=YC2Pb(Ql^8!O2 zuJk5ShAOThROpYF+q9FMP$9kL+reaa?Oq&|zPY7R!YkU#ne#lFll!!JMTzSDh{ncy zo-v+T!n70vg2#DeW`gT>KR!Qp^tDUQpqCROw?#V(+rKcMuC@8qH zvMd&7&hoUndLEB5*XDXjwm|Bv_>2P6HR`=_n`GUjvBEB% z<5U}r^IE!#W$^Yvy%8C9lp11obC;u9-)?0-*=6gY2WI=S)i-bXr0aGa7x%;n;6^zb zM3)lt*pT1U)D*~dlE(l9evXoWYvvn4n;A2S?o3nRqwUhGbS>23 z)OTllbIml{76Qff3;=KoW6W0Y?!@p#g-A&$sf9NtpjNRzQY|%lQisCRGrhf2M9TbQ zEo)r`$ea0*KJluKr_(m9q&wlgxtga=Q9z)Ps({M5BKeg@zhoH9-)?au#W;#AXu38a z`%6Txp!vp}&(ddHVqw|g3UNxn?Lyb#a^86D2Q&O1(a0BdOyEF~oWO7or!Irgf&~MZ z>-6g8T0ktPxkcsllLEE7H{ccOmHJ%3&Oc$6;*K4a0aMnEM?&ZY#jA4oJiFxO<>LZ9 z3ZE4WOIgQwodE!7X_sd@GtXNqDj~TmpUH2$K93Mo;tSdyNFN!pm05-NN+|Ds!bh#} z94aYHM$Vkgka-mXs}GWj@dPrZN~3+?@H~KvkjaNziu%19RQplZDObZ7X~R|h{2&QU zL;V9w%E-0CAcxn-P8u9Y%k3?<+j)eX?&$J2(1i=J9*t@d$Y%Q!s59;{xMZ#oy*gOf z=Rm^?KDa+Lbv2Q*^3SfKE(cpKERbpJ%QMI6VZh5*Z<3-_Rkefhk6M1+QEkt0?`UJa zfdij_J*_c~CP;(LWIF{3iTAT_SeaOCZ24s~G}Mw|<)vqiiWwugRmQcOIhl-D*k!G+ zEh_R*kIV(XBWIj!xMTaEibteFK)-k5^Ll;#9eC;_EnQGP2d(4n`=_Y|gazF$UVKb< zwUuSiX7y4AuM2<-T3A{tWM1w$)s{3pofEruu#G?4Gy!-`J3;6Wj}DUxHKbmhhiwvr z!8l9kg4a;WzE{XJ-@I=%ImL6h?;fXTuKHPl>=-#uq5Ws5xo>Hgpgz8DZLg_`*LG+O zqlwpsX=sG!=U-`5j=ZO+c+w!F+V?Y4cAu1Kj3|9&Y3cjvbf?%8wn^}s9Q{nMwKKd4 z)L!1IgO6S{S<+zDuV?To!6(8x=sqki2IE?Gv6!QEoNR1%F`}Ae;F*!E$R=xjxwYuv zka`;`^lgPT{J0!y{mbkVwtaudJPA}2c=Yh0MLFu)40J4`_>_TNH;(=}ROxmUjlm<}$^*S_pS3TFfUC&Fg5;Z;PSbfs-m{54VL6>? z;xkUln;K?jN%+$-ZSUFXija_}x#s6e1`)ZO`T|KX+~qPC^2-v8_g{TTN4TF0tbMOF zdy3I9xH7{t+RAo)tEs`RetRSKn7k6cb{p&%Z;oew7}r5Q6%035#te^9^A}8ZiCVV^ zS`V0+xyg9^`M{x%rCY1JX#J7Ai0yGrFjPPmo=DA|46nr{qs;?#baXoJaaa#LbpcbO z-7-T>+l6LEz}I1VOP{!WceZeKxW=ZthF1rv$xDRvdL_%KvnR4L%Mej<4nuOoJT&0-oxY@ezy;OJ+^G0B;I?v;eJDbI|Gn$6$kNf*Z zyz(CPzdNWfKV4~JLss#qKCF52b3{aF*3FJoMHiz^)g~`tvd{`2XQPX$D4m6cn>wGOgKR?AbIJ>rXyyk7dN33LZ8E)aQ;5lcSPSF4wVI%guz}4L1 z{w0@(!!@~hC-mHFkWIBkZ$c1L1ibKqA9h?h*kgY1^Zl0C6Pur!UqM?!b|Zz)D&b0} ztn#Oul5#kymnCk_n)KtV|HRHGjq!LbG0v_2;*hy{m9?EWPvlO;o@w72QDc~%3Kzeb z9<;g%!mbU4mV+^^*FRrHcD2M6TUFWxPe;9wU#i~zz-#Np!CDsPv!saK?$!fQZjr@x z22b^;)L}c&W@h?8rh;?D!Qoe&u?^gL>SSgnslO6!yEVL|j18QArsVyx zuLfI4aT*{i{MKhvDb4^heYigKZBV)XO z#wby{d3gK?Jw=Xwx|Nyb?1}&zGv-$Qx^7f-?wQnTXQR$e>Fy38A24PIH@w^TrXPl$ z0%NxFcOP{ksP?Jo&eDOl23B@)fm&KEk1F0>v^*DZelr<8szDvzIDX)$6lJJN7p}^? zE?^PobUrP8rrY+kfExlaT;)y29p)+J^14&d{9UK)ZvRn)>qzPN;47$fcgE}rPFGhq zFds2o?$+uF09W^OM9e5&?p11G-()G==r-cLoPygs(iN!3wx;6%W@;M%D*L1jPO z=}<GnMj#eyZ1DkvfYO&hSN9J?ri7l>-BguT03E2$F3Q>TO>DPW#_!Hx(dw}}L zmN@o^yZQM-f(D{5kV~+)HxVF8S)*v6jd*{oM)o(=Q}$#vmzg z7O*wO6@C18G2n3X&I93gu>4|@z#%~3cY*f3VGNK2;1oHPRP13WtS;Cjh`0Uv1*@_ zFY`=Rv-V))oVllwk=5r8Yt?ozP16aAnag%#Y0oAWPV(Ar-@12iqPjXTO^zOV!M1K- z|Kb~`+ebM$`y_-}qk}ewM{%!}w|F1x$#Nc33LKQmz%QV;7OvoU?dJVdbsOsgwVyi? ze9S9fZn%$BdL|{AQBqPS_DoMtYt?>veqSqYXn?_T6xbavnHv$7mMrNugo4m%Y2r6^ zr-Uc3o-~JufG9`GEK|kLnH_FTLxM5)US*Z9t+l6s=Bi?4mh&!J!gXL~Fo=rEv>1wu z(W2ymPj8OCy4fKv4WrkO&SNt}aPGmuJ5lmW7wUp1@2XL`Tx1otE~n=%T`zympj3Iz ztbf2ZWFarN?)bq-^AW!pj6R}fGjepae7(2e+#?Lzz&o4Gu~3%w&Y! zm5oeIGhVJT#BRkK=Wq1SLp(OvE`53BH?~zUcP&Y9iIpWb=9C~W6Sq$dqu3w)4nz1}~D+xlp%`i(PbP;SjVs+e5hLIk#pK5bs(`V`3_Vr?AGknz)=TmcA z8h2&L)_Au+%g)Y@nuuW5%Z^s#GPBxK_;i*&4ck~Wg4dA@ z{#P0?IXP8T^XC(Km5trmrmfMBZh`L(LDgMYyQJyW68PjR?j&;LjGF`|0Mm0I91+0w zte}_4Fe68Sl5U%KlER^OtwAx04(V2X?Dy9XJ!4gnyf-loyHPX;7&9?VsHJvb5QvAp zNgWPOsx2-oGiJD=rlz(s9rQ7!ZmZu}REekou2&7rF1#BC(Q>YM#PuY~w;h>Vfo)3^D+C{;kQ<(+4^?|ER0T4W8}ER{PA| z1kQjr0{}&NUNQN(JDVuQk+1MyKwf!I->1X(Og=N;?DYjl>12H)9Y5g z@8eOO)^FdQjJ@sZg2+w$!>JG%OO}$9(wQYGp5SJ6!Q31>jHu%cNY<&Wc>Td$u~g89 zA!PAxM=*7J$=bZ%(&~WCXhjG@RnJgYAp7a&dB2M{x+F(FXl+(jZ;@Rpe35Z~tGars z-CSYgL&{8CMEK+@{ky#Sz5A)4q5DE(o_N^EaMDteU%ued5U{rB;_8aJ_xOThdu!&g zyfSz%_=r%o_hN?Z)5S<Ozzlvxi4%@qAo;rf5zWnTR!o%dxUXb~;)u^h>&fq1b8T?CdNib zy6b!m9D{0mv$K7GDc-BhK(vkjzMgwskbSwVj_U2iCmh*=z5|bnii);34}Mu&CEHIw z*cvZ88S^9|Az_$RSV%;K)q$E;+0sb&+5J~5Ga_FYzMN!|13b(xe&{?B_K{{1b2#>1 y1pnU6*%t)})QeQ@uKW+Ou>ooT literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/ccp-games/index.html b/content/ko/case-studies/ccp-games/index.html new file mode 100644 index 0000000000..8867cbb323 --- /dev/null +++ b/content/ko/case-studies/ccp-games/index.html @@ -0,0 +1,4 @@ +--- +title: CCP Games +content_url: https://cloud.google.com/customers/ccp-games/ +--- \ No newline at end of file diff --git a/content/ko/case-studies/comcast/comcast_logo.png b/content/ko/case-studies/comcast/comcast_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..3f0ef7664568017f8e49592a08b4e83c71964524 GIT binary patch literal 6279 zcmd5>S2P?@wEAbK}?FEd0hQKBLhsHdz|UX zmTbccOx{x8dAOI-%Jn}QdHr$g@2&Lq(j;BNE$TS?2fuonb)rdxRU76N25#Ik{Xytg@59;f?MYcACj>~rP>CZ^SJJt;c{4QZR`{;yEf=F@a=NO4P zG|{qp$b0_bd{EkV`vUag1LOub@g>zfGc1hU4)7Ampps5*qeXz8a4fvO z=tKTGlk;cGpQ`FdNIwyCyFGNl#3KLt8% zPZmPtzcI1RY@+T1Tk~Kp4HjYkH23Ot;3C}!P!{U$549zi`K-0T_O%J3a>KcJof4$r zj5?G~=0gSAP@hZ1noqJg`c`vJ<=bRF6sqy>_wV##ou%s?t~W*?SK( z5f2PZ4ts0Go)i>b(ax2=exaF(5PJUQXfA~Z1s_IOT}SoE%*^M{ce`pi!?a!7o8fs7y6gbVzX1zNn0fC-E!m$l znY>8!Vkng!Bb6&HV7weJRwnQcB`6jpwy4yh?FF-~**2~_wj#k|ddJTmyZxi+>gCC> z;k%=Sp>{j`t`?)wULEfR>p`|_l?6d^>J*7`9ab@n8zBqlJ)I#|t4veei<%v=y1_gD z{~l&;la=b!2rclUT(ExmcFmjm!94N@y^Bpk<^|Bu(fyH~C$Z~CD;v^92MrTkA>#Fo zR*cx27b((ls{du0Sqno-G7ujz7%0Rx+GAw<2N3e-7blb9rsV>b@6&ANa0#610sZ=V z2G!I_KC;29;R2&f3#sjd@(7&MVg|x7v-);D&s%YI7nu{79ZGt7LWusAfK17b4GrNvula*6GWzSi7jnYj}!+%_)UXH9e`ngb{Fy zosy>w-p9|hK{mLk^-uhG^)u*Ur1vj~PGv`*MKkinl_>pqSI!e>vc>#qrVi` zZiF8ZUj#5KB221V0)=N>d}@qdZ;PhsNB#RHw>>e>zpUgi^O+#%x;aVK$J{s^PeYI7 z3u@DC97<6c8Nc}tHWOwCq&yif!BhM*Z~}yiS-Qquz~9Td25E& zl6!}xS*xtvD^~g_sVB8kBwGbc6QJ99#Fi(~;f9Fg6{GGNTjgTtO7cf;g`Q_$R&IX+ z8`*@xv77Rgsx(eLX)U|cRn2^@!hF~a{W1YPZ6wbumbrqhvx&FUR<^KeMfEVV!wL9J zmx=ld;t1Vq1gH*ht#?ECzlle03^(wz3^+XuVAP>(`!;q7!z})F%`J73xFc4g&tiF3 zlp66_0*J0fUduXCiddOnko@PGwd191-|0`h+3D!zkiw$> zQHApIHE=BEn8cfzgg?G^8>6XMO@-v)^%7Kw@NYZ>1yBnjrGwr5qV;s@RX~bN zxx37CSMh+oL|CSB&tQjS17um>kdul|oO-bE zTZ@k$R=>R<^1ruJ}3sM;^uU`~G*Kq|em8gA+a{r3)mQ zugmPEXEUrdW!GtZJ(|Kp7{=%{_Yw}z1*v-p@mj=3j@U!(MyUZ zbG7Et3gNLU9ot+Z^z&jD2>G0tSnzu3#Hy7(-7%-qp(i|7ibKW;QrO|e`RWzM`>|BT z0)_az^;M!aw`Ew&dma0!dV#aewue`5?K)@~5`v$hzc?Az!MGxy6F459->CAv`sjL# zu%CTk1*X)U2*ptFaH3|v)~tPZzn9pW$gD9dFA?+n*L@Yxxr;oOO0q4_RPFuC z2UUvxNFrDg9N^m`uNe1@U*+oiq($&if7`P9K%PHpIG4g+j<~{zyv}L9d+@<=H zf5JD}aRq?8^i0`8@cO@u(|(jsI9ST}1Y>`$S_w7z?IbAA9HQqH-XDnuEc&C4tQLN< z6dn>d%~zAbn4Wcr(ahYu5b(l-DP%K4&{B^8nt2l86>nK%&(VtrAY9xd)uhc8@+)Su zu-j@$A{~+XjM&$!#|1x@pC-A%Wz6)5~?Br;0 z@S*#^YoQSvI}%KIoVB%*}J;Bpi%MkGI0~Z&Oi3_x!^mR*tj@WKT|4TDNV|2 zhl^{}lC5QNnl`mxM=ljoIbi`~UzsD@0SR={p&Bpet$&SDrde+w7(U}CmqnpAiM%}d z;@`bOmZ-{xN0n@4BaFU#1<*Z3Bl(cE#>j?ea|`RE&&_m|7IUWNL=BfVz1VRaJ$%l8a?0 zsygCAv*m+VNZES{lZ*@p4(zVy?muV6S@*Zkv16(kPH0KXIZKAFp~dQ!@^6Y=77lbYzI zfaGV6;4krSSM=lK9D>>n#FLxE}|OEH|1&rYQyU5u;b#F|hwWdB!?A}>%zx7k7D z7wgXlG=aDVYQiU)_X*38osM~msFo%Oxj2*VfgAk*eU0rxvYv5!}A_Q6sg z_>J3K*pQP7Tk9F87S`ysk_`S`lZAcivt2yF<^9<}97f>>wkxw2>b2$_+%n7n)5)v# zO*B_K*wjmqE&f)u`Fv|4R>ykiJ#vdBIQp)(Infh6F~1I$bn6K~3Jlu)qn7x)qdj`+L_+B*tUS(UOq4Ih8!LDM%VCoF{W@? zm}mIBR)-Xbj7r7~?lK4UD}90I7bR@Gh;~BfEI{g+W?Gh&v25!6(zs(t<*{kn-^_x( zefJJWfRF7?VK;>cfY)IxB~Uh364DnC$$GLLXEM+dv}D{+qNAG!Xv`wc;!sL_L5*Zp zZ+;-&2orYxqt`t0r+GUY!~o3x@SPvo8f@k~;=I`68EL;5N|sMS%z|tMK4^UC@PK50 zZGY4*me!WNS@0?o7-S+W^oIK-c@DYdLJ1Iqrgx>%+v{{=yZwjv8Xv5J} z-u)D^+9eKgB3ueC==Hz4+2(O!`gbi5YXKp~zuHUXD%6lk==|N_QUyP7;?a<&J43y( z*~m-6KO{Od5AHuhC(z13MPW$Gje3OOH;N#=M|90^g27FYjx)*u0~|dGkhX)Gfz^Ew z2WB>v>fNAPZ^&q@7i{C8#bI43ds=B94`hR56Tic%A~V4@%z1ZSnj)~I{wA@8BHjr|B+ra``(8I!`SgsdF2=qz zXr}o6&Ce1h*c+g8QR##YpLVmoz}H*GcsdO?&ZVwTZ(EyDTX}}9vkLe-k7_}2*lK_k zBJ3y266xJL4|rYKbByW_9cDia$UN8Gn7NnPH)Iqf(A?GMO5kz@q(4i2QSn29^IE%8 z&Is(G8DJrScuJq!FWl)&9byz?im?$Z6Q!q8zh6p{+7IAX`kjsB6YBVUd zGOm^_8`<#N)wPW>rLe*ucYfD~tN4`o4XTOBvBLKl>UtzW$a2FL00eI^Urxzv!h+E5 z62>A9mMS$;@sBfWBSC8Ck|ALfPM<65DbanCW;B_jQt^q1NL_Y+4{flpa=~hxU4NDK znkzF>LNBN|;m8G9a%B+}#}qNw(S5#+7)z62k1EQBVlVBaRH@%8c1SEKo3;w+M)m8L zv-^*d08hpaK2hbR+V1sX$&d*bCNR4Ko^QU5y1yOI{6x7U`rd#(Y@GI+2j$P(*r9|> zyXW`6=cbY(vRWKtTU#8;BbO_xN9UGt)C%@sEn6IiQhPwC`DQ*@<^Jo3$as~!p=pxL zq1SkoR^!|he=S&wjnr?7na3}GEmQ=klbI}>l0TQeq%AjK021Ju z#-gAQHtVqs>(9geLK8i2N5SGvRrx9B^qgBQe4_a{r|$J9y$_EAW&u*sf*?_u{sT?S zG{O#PB}4h4I!njXs_+*~uB3rnj|lyl;ZBBl11qTI3>E&nT}=S}wmc?A23Xp^v;I~QP&xEa5fkWXSUEKm z@|c&O6_$|BFVuH2UzC6;D8Bo#Bx~_Zvi@P>4V}B+5rrpe$@a+xFV@XBJxv`|&lT%) zWs$4vSuK=`q!#f_WchSRptp~b&ofVhr>@@1GN9$^76}eXg`< z7LL~Z{p|y8uT=%r6;>{Kjup*6th^Eedi82W;w&ouUi;1*o9?)(Yd(h;xVDoS5eYJQ zZ5;e$e3S=21$NdhgUULiY{dyP74+hZ4E=@dX9ry29YO1hKzwiexzZ9o34M!7-Z|&@ zAf+`X`-&ddO5KaoZ?$gt3=>8^PFwUf6*SkUks^vq8_#v)qsI>n8oNe}vRa%uY%RVI zj-_V?8aNFn9LyAT`s7NAR}a5(R~p99zA4L_Y#e?0aaF_}z!yIIW4lNx_G^x9f!4p? z1e70YynZi-leXuZ5N2JAt3$&&kVb{0zJMVNTj>)u*;(#V=K(GTyN;^LSiQjn&(N#L zthvMmMbU+F%Xz>YFcqkO@Vy-ZDV*6H_KF~4u;M^4apixz`Wu{sR*Z-?^s}F^!+M%; zhvJ3sP;b`RZ@L;^NYN4#)KVGE0JSc%)B|g>glBplG))POO_8Nw^3!%q6L_cI4H?>1 zD;7g?)Nf!Y+WC{*M6Q{nz)3O*kGBVQxQ9i#DVY0`)oLpFb3CnhTt5;#m7MXKlQVPo zS6N-gH$ljJ{2`OFhyJasgi!1hMBezDD!x(Us)JAdcX_8(@4Y`bs|*3`eHE(<<=RkF4lY_tP^&< zX}Hb<43Qod$`P~5c^kyNlu!Ip6c83!Q|UL#q4P71Ad$JyK+IclF?0aT#7yq9vD45f z8eWqvy1MUf9e$h+*k}q8w0~1ckP&C;ZX|C@B>l-@e)14P)!}nVlaxdcZ9KX`$8xDB=6aaJ?+^S^3%vnU}?p5Jako6pi2))T23G&&|ExxIdGTGPBnE&bjxq%HD%9Li8zv<1CB^7y$qP zi>{8A5$SV?^a`e@CB4&H66Q%C4DLE-Z~y?)p6wS6;A%P$05~dzF)_!R>z{-nv93}G z6xJRs<>Ts3QUd^RRUdZ*(ix5Cw?{i-+>}8}HT58V3`!YfCZ`Y4ch^KaVRZaF(8hiS zCP+VLq#_EWs=^5OfsqopqVWiRA6FMQ9Lz@<^ffPx6mK_!LHu7OcxPph`gTEnbNy5N znph7sznql3BoZnk&99&+1(j2jJ}N80FAagpgCVkDsG=lP21a_yLioQwKq`!+2=0M$ zfEj5W`(BRpO&R2b$GgM8U~g}4DQ_7mtcN4XuBZrxNQ0%NB}ocNoUa=m;UnpW6a1Fs zuRL049MS{hj>ll#__y;S?6IDBWe{k)(2vLW^Ky0nQOFJVogIlH*azVbhDt%ezcZnb zKRoW99xh)G4TS`wUC^#*H$0BygZ|d%?u5l-aZcEODEXuN9}APlL|^|$?cW}+tLtw| z^z@pVUojnFu(rw0$HaQ>EQyXPN|9qohp2itbf zH`~{#@NI!R{)mYwz=Cc6W1xgMT)XDSyl?m?qW*>p?m! zw2BNI{JUl{*Voww({;n)5pGDdu9ga^K?;LG!5okfMJN&?BZ*d&M@Y)bBO#Iqdue$| zSt#1UURD;Rc+>&;6{5aAx&2>KYhjU|+biaKYSe$3n#}$$sSQ0aq}>kTLZ;b1`v11X zZ_@lrDIE-sG)H~O<}PUlf8EcF(dYi&`d5bw=Ig%Xj_|;tw^yAq=&xn>VqPi=d? zu3i`dxxJ&QAh#C<8U+Xct@Y<&{cwJ_VgA1Z_$@z~{hw=iJE7hF-Gcg->Z|IXli;up zcyELUTHTQ}C;rp*{w?$0t-qFlgSW2&E||X$#<$kLkI`SV_Fu;ozP-7@Nb`YoY>2Pp z0EL79rR`_sZ+i3Xadq+hs#lPPkRD*@_WO4$dH1i@GymR7-uk*oWl5^UCgyGq&1A&o|)26`R}G*dw%OY^V12H zRv>qh^?Y}J8!WO5``5Aj)=IiiX#C?X;kP;rsdpcqqeQA9*8pyDE@ zK{2M{qKJrGK*dE)gJMj@MG+CXfQpNp2E~|)iy|U&0TmZH4T>=p7ez$m0xB+Y8WdwH zE{cfA1yo$*G$_VYToe(J3#horX;6%*xF{kb7f^AL)1VkraZyA>E}-Hfr$I5M;-ZL% zTtLM|PJ?1h#YGVjxqym`oCd}C-{NBY`6EEI8|jyS-lU)S4Po~Bl71e@kJK^J2LSwp z0f6(N0KmsDq|dhifF}e1n6(1{V3z@a{n$&kO~(L$y~es)>LxxT^BKNArc)U$^-I3p z)s~B$b!xN>LNprY5_YKGvts6jPGttWEGnf^#6OwyIXk!b&Q?qgV4R7}$SdlwyFE8Y z#Ltc~Mn!2P7T?(Qm>$6}ydG8~~6^RDq9| zC*I&1(??sP-xj`4aPO{*fE_|YLJh!NRa!ujQFBCoGG}6(QHXXO$U|+u^>M@uI3s;b zv3@3oL5gz@9@}zlB}=1(h+E(6&I)}c9pe=zzsvBp{5fG^A%3BqxougGvYACh^{!^@5hs+lt>n81aU8GIzNoWd?qLlQ z)}Ui<=vui`oM-U#NlK{zjIOcv1%#Fkm@Rgy1BcI@m2c|==;p*}!ACpNV57~*9Gb8= z-N`#{A5XqAO|mlmD4&*c+-o^pYemh@@fvnAaZ5SQCQMl3#HF6;)&s;ud!A^YyH~)~ z1994SvQK5qc9%$gIcA+>s^57ggDdau18Rop4u_3${G_X zG@&SHe_~WP+bqq&_4C=M74}5MFAa~*nw5lh`}eU37^C&X3XjVx?0%Fx%EJik!<;^0 z#Z{6{6!lj7@KNKwR*AKkm&Gw7sj@F~tN>;K{{VzxL+P!e$LKQ5lkr(C?gHkuFB*?O z%m6v3km;7#V%{|8Lv8B!FS5B{w1K6}*X%y{<=Mx7dhm!OQj<} zRWlA+uf$BpdpCRg-|nn>pLaOl0~1xL*Zw9=$#~14U+rCUT-<#r4lcI9I($3;`eH9vX)a z?WZ3#wYlP0UGZqOscks?%xo~<%ouSQa8Mj5eVrgy{2t+SE^^Vo(pN7~H7xAv<(33i z7GBr)@YS@;f>eVCb@D9%tq<28*4R34wTpp)4f`MrVypa0Gb_CCDeKz$xn*$jn5mhI z%Ju$&;O7(JDVi@A2MN>3Fo$?u#*cwPwROWxyGq}wTVGZPXZiT<*zOx)lQWg%=huTwLjGl@^M941ky8v#TnS1ex3Fr`v!Jd zw(L$ay*^nz@V@ku4Ta6aPSBNY- znTO;OL@GXevw+8B-;Ug@F1sjm&!-VuFT%`oTqi`g6Z4As!g-otpcgMtwyv_QtQ)-b z_C0iY&vMkAtF+B82eE-~Ka`{xr*kz`>@2U|W0Cj5y6Me=%>k zIL0>ZT||x;`^0gAMF?F`#l5}fX&K_wr|dL3KV(Aa-09-fJ>fm8F+WGZoa9R%}hVT3RS}%o8rUVA1*xpsrX~~^g_bojE{@)o+9!26DPU`$M`lL z@(~cHvzz7f4C{336S)fp2{tY+ZtEbuP}|#*-9a<&p;^W5Mc_3T-xD)e1gl;P(bPaGvXQ%PxF*LhBv@ZQEB%i z-0HVl=(-!RRdlro(RoMgT`L(yfAs*6gWz0fvzOg+m04PXwRK%KNbf}3d6}T<#oEkb zzI9FGm#ihjgXPorm-pl2fn2sFU2-Bsf5L-=+NX3!b~{Gj;0|7r-F2&@2XF#A0ilcf zlVjOC@)O%ZTw@N%$?A`!^~l?%Q$xA>xmGokZw8N^Z5cAF#m}zDF!5csO?I6EJX>7C zxUN3T0ogr=*1v9>J2DY0Ii9$>F;ui`(z-oMiH4zJ*>>H=$N2ZZggB> zfzTA9aD9#6uHhllzUg*Csnc@eS<4YM!}*3j)qP^ih3o#=;_RoaN3iuq99596L*-1< zslx?LD#UE5LB9FW=A78~WoHW0uD8fo*%Hdn2DK$&s~&bGRQjyn-B3|_xG(P1GTRGw zJ7z|=;No*$H?Q4qxSD(uBFxJX>|e?kw8LxPCFN?A#`8x<%bOhn-OXHm9>#qJfEucNK#`MvO6r5f*lT*wh!}ziXvK3;@EgdJt z@BlZ$)Wyz~g6^6tm_otFk~Ij*mC=I8({je~KwEW@q!9OjbVrOX)_*B`tKcx3EbESN zR@2FA1AW}9H?KhHX<{vAdN@<&uc!@Sh3K?DP0-=A53&@Vzj(?#Kf){Det3_D+HT{9 z55AM-$KIbHbkET1I-L(uL`J&e3_7Z&swH;p5D-23l<(w;sJsu1%Z=S>^3{!4-o?ReUW2|0to?$(=G(^2LxD6-r0WmbbZyreIg!+$bB9gcx`4u37D^*liKVZFQ34FRUNX|s znWWyZJO2t6g-DE|F>oba(fM7p3_CCP!{&L=uR1WVcCg!Z@JR>uC7NhyOg;hM=CK`} zjZ1z}IF7NERumCAuqAw%^UUfVd6bUUmolkQTSenC4%ue{GFG%N=i96gvG?xLq!rH= z2lnadWvq8f9*|Q!hT}HREwra+zH-$@+ly6SS0WbI{|*7zgIBM-RwDi^$$?y^6b@ z5_?tBKI)f?mWuTp1rih#9or>MwG9?D_Chn?LFwvy=NX!+{%o)861}Tw$a;a>%zk%M zaauCGD%az5<=lkyn@y&e@q1`a7r2EJ+jyN=ciG4tSs*9lXuxyjOM{jf*zkZFAQ0XK+7vO}`VQr*5ZDk)`)PcPe_9)1nyVP=S{@1Ny< z7n>LuBt5d6>LHqb6C*4nlr4Jkl8EWiUX5ENeV$`QxlR4=ueG7n>ZfGFmCT+E>>WN1 z)rrfGI#&f`mww^Od(z}Aqot|YPUZdy*Odf~M_qh%ptlE_pPxH(pib1`T0{(NVp6m1SP~Nu`+P5y(-Ki zw44fV@ap1+1eYAERgZo)8&zf8Sc`Yvb6Iu>@Lj{_OEc)&^JVkZNu=ARX-&FDkEezD zH9XHtAsbCB{Y-~ja_RN1?!ETKfN<;cbOeL@4TznEdE3O)e9V=P!pM1grlF@#m0&?- z1xjcg<+2*05?H~e_SKN-91m#6HSunn({_e)3n#c3O( zJ+4MhjQ(rQ(e0kqbCW^lKp&TpOUw~FQ7F?Qtd8o_rL5HFc1i=UA}<9j+2*@o)T`&V z+V!%Ao-yubW1byU!Oh_y@n!FlLwKQVq19Dth#BvRt+#pa)v97|L6(#LN^44j?u)zciRtsg-`5Eq1O!ypZH?5 z(VtC=EiD#VfupSJkDj1U6$MuzH#1vvSs3CT=0D=|ni{nsTA2D~3TuiH?&&K%^G-M3 z=n@-e+UA{qyU*N$Kzz)yXEC^vTb<^JYR`zCS3WhghSUjVczNoudlEX*}2%?KU5f9d#r3^oFXj3Rn7dTOi@jp z{lfd`Nu~vQU7dawDFGUR)WLn&T2G_sBRR?KPi*U|XA_0@#Y?cz?D$h#d?$ooh<~c@ zOcSy0Ou1cvkr!}4ID{?&zmrA4$hHGqa!8>>f~gMjnN}SkwB+t>z5EP2&{6#4frfuI@C`Ejl4u zI$BzR972~$PQzNm1L(VOL~G;f4%P9oUHl3PRfUCfeR|-9g$2)}2eXp*U&DoND&Lm- z@@dTA@hEaRAK&BSer~w_%HhgaK0cPjdaP@NW}Me97H<7=Szb#xXS=$l{al8zs@W90 z-F9VXoGVLaMnRwmF*NjkYx#R_;k$GB)7OjWr2F5pxT?&)7?mD+%{oWh#2VBKB$_6b zJEozX66(jNNBsucO`NQ%=PrTLggqPh77vb?ETzW0wyH+{q27F~Ek0>+K_qas_90EZ zoAGcm?P!YWux(KrZ^>Zbb?2Ely6zmWV4|J*iM_Ca5zr_pasPfXFTX~f7fU<%!>znd z+jw5+#HX}xJiWlHslj;WrHPr16>8?y!e|w7GpKneS3A~9YS33=qV~4Jm$POkA79Ta zQfn}`G5cKZ{NzOXn|s1RPn|pG);&Q%x+g$o_j>~bZCi)@92bVQg|;m4)@Dx)A&ESd z&Yk)5of2`6UY^QQ?NRn0>#ZdE`R$H$eVw_?leAKFYA^j2Um61=k7Ejko5isVt4Wd~ z$3AB6GX}``zdhwo)MWeoYJkmqq~vBt9Xoi#b|Gv$?mfp>F!2Yh32m1$GcF z{orAygT1xukH_<7t_dOh5)Le{%AHR>MUzsaoHJfOxJg|1Yv4AaYtCj0+qqY5vZrq` zx|u%HQsZKjZeV4ydCD^8q;&rphu9g#(62rqk>no zQkzq|TeHm7K$oI+E_)LMId?|uT#VYO4&oF7sgr_e9d=Rx`tO_npEfJ=&r5)&iyQ`1 UAM}rI|En@xZ3C?m4ZD#41B|(?>;M1& literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/concur/index.html b/content/ko/case-studies/concur/index.html new file mode 100644 index 0000000000..0bb619f527 --- /dev/null +++ b/content/ko/case-studies/concur/index.html @@ -0,0 +1,4 @@ +--- +title: Concur +content_url: http://searchitoperations.techtarget.com/news/450297178/Tech-firms-roll-out-Kubernetes-in-production +--- \ No newline at end of file diff --git a/content/ko/case-studies/crowdfire/crowdfire_featured_logo.png b/content/ko/case-studies/crowdfire/crowdfire_featured_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..ef84b16ea06c7de72fa288068f309fa91eaa36a5 GIT binary patch literal 7827 zcmbtZWmH_vmWAMrYoPHaSa9z~12hhey9H?^NJHbEK+xds1PKI4LXhAR+zG)#aF^ij zVaWI1_hx>)H8X4Gu61wis=N0&r%s*ot0FaE3itpZ00jjFUr7pcWs1?yO#wLjv{T1u!J)xIa%1kwcr-kKCb<6aTF9ZUwa(`cLOz5Q7eQK zx5Xb0x3`nagBk@zT*}+U!pZ^e&R_|*wRe^P9W=Fp80@VjK>9*zylO78a65ZNUnE@H z7p7z7>tF@321!XWh82+NTJ4k^35tMsS5v1nn5@Nky^Jw*DCEjYQTsr}2av)ext^^h_iZwnV5K5kwf zC#OIA`it%6t_A2L8v?*4m6e~1s!h^iy)9}dOB z5sI+#aDqF#D?ued4}Z9=?X5)xt*pSpyn+^7yu!l#T!Ij5K`skkF#kgWdHLb|mX>^$ zaLC_w{+(U`0)`6l%g71v%M0=G$$`OgBEqsV0@6YPvS0x@Y5BjkO3rTX7S2}izkS<3 z`2MFB`rm3rWsz_TcLY)gfpGl03N-8x?g%$KgbRbL>|dwOEyTd4W?^OT{AZc{&r$tj z=TJD(-V<&uk3={z{FQ4_`+pP1|1Jmr&|CjsbI9`$6wjZ6`7Z_Z_tHaE{(=8$hlk0( z8Vc_G&_&3Hc99^Vib6pFRwzNGb-ZVGO>xq7CM!RQ&R3s(!0W{8l&mU@2!781Enq09 zM|&DMJ5Kqy;H|j6iJ68D-qNRMeF9H`Odu$+;7jbj*oeMXKo#K-O10NTslLZSPD+9y zWl~A0)Lo34*WzCG{jF|LjvM0h=g&lBQ9&Xak?%D?Psx;_p(L954=6bT@CQM4*d7pb zfy^I7O#TNw5FgM3@elexNd71BZ#5`hC@oSG5uq-926DL)cd&n+cw@1@Ymx%YY+!)|o=)Ln_|) zw1p7MvpUXTg*@CKek_utc-hZv3{mJtXvg&KY}+R}vp9`4JR8guq{MGB?B8E9T`|;Q zH+?A?7kC{DV75%*{i(2fKwsl9C5x0mp%-NYhU#)-v;+-gHOmAt0tqH$pr?@NPD`y+ zKiP1%yL$rhWN#3)kd+&b=(-i|>SU}cr= zk2$(ru*H11NkKC;@VjrhfOp!KmpM{Ukb}6~#zK>L-jio<@eMvOg4R`vOsPhOlSdt$ z4KSlY*@2#rKuO+hu7pj&;S?Hr`{S*wsy0c9DWdaq(o7Si7uq4?@6&ABMEC25?VTIj z`f+b1>nB5QY)g~uhSs)Y#P&ZuRSW#bOp zngxUQ07G-Q#0f+M(#nlaiw(^h7e_LdjIW8IIYHv2v z>C&JSHOMmXRe&>I4`x?(eKV>^td``UN;)%tW|t*{9z{VTK*` zoidE!2`*f(jMj)?BivaZRDBPa4Ip`dW5!*bSpUGo8%D|g)8^z)jVT3IYol#XRDD8P zO+h#Wn^QSK81RUQb`Xt7_C|oJFy}o1hQ*QN&`1%M*-HtrEky!h2ZDT%k+}Zp82yoe zX4f~#@Lf(-MU-%zHG3-5YF^2yX2B?!lRi$s6*=v;C}rj3C%AkFM*E6?g7~-KE%}R? z>wEv~D{eJkm^VFvAJ*xB40^_m-4BXvQQ~lMr*92JmX<+~D1&&`!dT;}%~kogNa9_q z>r>MR6?7J=YF)+EWDIas}x^hqp3RKm~U-340fbp>QB8>+m0s{s&Q0wa1~%6?3bFvJ>G ziJOLKLx`GiSl2eCKNE3s6cB<68Ce*{jN`Ei_k{znzK?G9UEpRM6Eh3&=UyQUb*&(; zb#blVH|fvh<5rpRpZF3cpF1z~G#?WC9}^S_7wp_03h;4o8ktvAwaY{wrI8i)XF!)- zZRBGw`6vd8W5$4wpE8vAot{EnUwc$TR?VUnO~O4Vn)E+%VkCi~`({x)V1_SQ zz%<0+w?eyiJVgf(r=&ko(=EW+@3m2m{CCs?tniCf{pzi~%9_BCgGS@smT$xg90!7& z^@V`Si2;`v-B*6%3N@?0eqs3ZJ>eb--!f1x;e#Wb-w9aQXy-UkU&ZxqW~A@dRPRh@ue{CzHVkn> zsF0*yn=wZzbk}_rKiL=?ND@{;Ht3S(t9U+qy!k{yT>17Zw{geg5NA!J+lp)y4BUXY zIfU9rWKbGv?B~YZnSNS_ec1G!S|ITlaIm$C;M(`BPBxF#b}Ow`7YQ`=OoCKnabFLe zw|(D{0aYVd{jYGKayTup0}9D6<^J_fwEkCo7i+o(@$8e z2DS6u)PIeJf(n#Wj^ zqmq_UycikQs)b%X8AW7}O3xgpCIQ#+7qzqd4AzBI`yvJ)Z|U5FovYJMPeb99G_*U! z=g715Q3s0E-MFoZYP+uoD&Y#?C#=m!)F#!@1ba!%QY{eoz3vhUvi!7_8p)TyokA3<2Beg;+1&$J)^#W6dJpexDoc+MEN4P2(i$%(LekPafG0mjMZf^}8>W|x$x6iuWGf!(w?>~~^gIBFf zmj>@Gf$QX%W@$^qolYeu;+-BY>fXC|GGB&3^d&`gl3K87mfP=0qn<2V3aa7}p}I6` z-XNfSImMeRF6=CUrZf`wVJ+WN`6dJ_=1M3iI^|?L$edTguQQ#na@}6g3s?`NCunR? zzFfFb4M)F<-|p5o(!oy3{X$b;Xyx*ic647gU0|~H9!N&c`>M#%by?`E&*L{cc?v%f zsWR?lG~9v`H-+EgR_WfVvuY=F_r2$zNq&=LUL=xNQXmr@<29~f^d+99%{fz~v$tpV z)8kw8sCo|p(W>xVJFy@cIh)a!NoCiJKCue*293x0nBY{k^2f6N5;n-*Jhzr-eG!9>Fql8D|r+n^HLdxnAeq6Tb;7wZlchZ+MxQbOuT z*qM2j?P7IKV4PVpoF=EpFT!aba=YgiT2WvoGKpW6qE|{pK*P z_EUGQs}|MQl`GUhQtCKy@q$9e_Q+iJ?LKy)8%Rc2j8`@Gx#;v7+1wX$aCxL8rkHT% z5ffL51pZj}`k}6vQ->sNxEWlZYfXZo4y{g!?a3wuc}Qe`vLD%gLv~KQM3dJUZrvmZ zL_4isVAdnV?)*K>36MLUGSok@R-e!@l8NIrZ_t1H+@r2XZINh)v@fFfx^&_w+b9>? zYi*a1%T}i+LlE&JrO_Nc=1JL#RLQ8e7eBoF!eze?IcbYhvx`zr(UJI*hALm0iG^t1 zOQ`jl{mX%>=J)9qo}s+<044Kpy=YDBolGFdMFy6BwdQm|T-&cN7u(rs#X*5}*|D_5 zGOq#4LnkJ|NG(Ms=VaWqHPxPHj7Ur0ZG5EUhhA#H-WY@*c`0Zxd5Ogbn_9lOxh8nK zD_diaKOK*Uq^KQ{comKwTI!)tQEZ@_HCoQJWmg-)mUn`o-!6wXuNYAw)xQ$%zpT1a z09uw!qeGHS+0(uACkuKKQ%|*Mti+DLBPe*Vz+7%y6tTyV+n4kqJ24R!`jWY|4?8y;mLn8@=H=v;=z30s;k~^ylV;7Vx4&L*GxAGLR(@v7AfF`;F@n)6fl_%Z znTZd`{Xw|zGlQ%+*(s0?o4=L~vEQ`$P6rz_^CmLOV%N=A!R&0_CZdvSXK*&*+L(Vg zQ()Mo6p&21R9}7`h&bObvIKRke0%lf^+k5_Kt^g^e=wEj4W_CSr;HuWfp_q=8OK!D z<>AVXYp*;|Qe)!1wWP8g*2K}A%bWB1pvSHJ!NNftzj9~N=>kL3ONaDD z{E(ZwH%DORGu;?A>iSsEy>;sOBjzp{Rly#4*$}7&1S@crGg=}M4RRHqMd*XUnk$v! ztx5B&)IuOC;{Dxw?9k7vkn>2XS$Qfg%#~>}0=EIS*4I8g%T;^Zr%P{APXg$UI6EtG zQkgAIZ~Er7XtT1=~4Qkuwo4VR_=+L*&o8Dw&im$0|Q;Tw#2IW4R zjy;7>3B*f>@Tu50k}tk$iTDBXLb`S&^;Moykt&$0+BR{0n5jx7*rh0AE_$AJvTi&w zWYg+klDR*_X*PL~rn2RdYTH@o^peZLTCqNbOfPL!y;(tOZ7$hVlliP0yhId}f*D-<|?zAJoHZHA8uY9#=d(<04~U`zhr|>cM5`Ppr@*=l(dfN9IV=u#F9_ z=LgPpLn}jrkWjWxB?rmwt_fiv5St8<6`-RRlu%7s@b;@Atmo=|`$Q4xm&aJ`rz~bS z$86@pScrgn%n-Ceoo{ad&_j|z^8RVOv0rjBkM|Q>uLiIRV8Oq-3HX93hB9Xb{BSrWFeJ?$9rTSwqva zS^BB&`Q6$a9z8i(UF!Cc*i2K|1SzQ37Wk9tbwzs$w^02qYdaO}bso8vmHX_LR4)X} z%s8Uj8(4359!^DDjJxVcf2ogIVSG3`U}uaYw*K^UO@Q~U%QLo{0a0wjjFBA%$ID{a zph&4bKj{j^BKt1)MQ;SQQ7yDsBWFWQWip6(M>$JO0y@j~=uvLEgG}6+bX7um2<)T0e&$IXPG+cC^&k zlqA8Z)&{ZTk!9>@GZ}T*j&-p5bq9t-&EMU>3wr)+gRna##0&9ew^n!jDJ*)@?vj$y zQ>aMT;^Ns{DSdRw&i*ICx7D1>Qy#TyC4e#m=Z%lIUd6`d*T8w)Wt-=3=} z)=H46Bc6QOX0YcmHj?c=NgSj!FV ztmy!S(BVJLuQsxNv$P0N%r|5=(^%$kr0ROMhRKN5yUy_mBKBasiQjBT_J`?x z7aMo#NfJxt>;<2Ue46G4JAjMs6bO?VHUK|_wy9=nE02%6Bi*`YBH2{D>m>Qn`o-d+ zgK^ZU^m+kKRn8bw{yuJBM#AO`AWco!Jl=!GEx6H#om(Xi2fFYoMr(R%e56-OGNM_u zV0U5V%CzsG{Ce&-;c#xz+P`{leZvCB=Pu}QyUCxvEz(V@1MW677w50q+QeN9(XQ0A z^4%cZ2oZ@F7%bZ{R-|ow7B0Yi&S#2@UFKjKit^c|;gkBY`_rN|9TO2l%y`O|5o?j> z0EVUHQkGA=oMjj3jhTSyLS(dLFN`XDYqnzGxx9!AE!8W^7-AjMU}v ztA@aCzD-6q!-^LK9J8M@e9p9i$WlSEa81366mt@VA(oQo?s&AK&#(efYO|eF@}zv+ z1Wi2Ap6g8hprz$iZ;=!6OuJSz5qB~sBs2+Ejh*QE9MG~pW$R<5>^hPg5ADq5VHKds z2?}7og$q7m@$r9Nn=@HHVgE%84S+kBB6S?jK7ErJ#X5RtIlcqP6fvdvCj=jX=3jB?y!@JBJ_7d1{f^L4mM zT6VvlB#p%fiba%dL&2W|1pOVVJaJ}3tOVe?=l?0yE|EzBnwr`J`^c->^qurh+ zL(UF2gbY~v)3`ethfG1faI`jc6p4_ZCH9ASZ#NVg)&$k{;n>E{S=_Qn#rY908cS=U zR*>i&SEY|t*7*aax@;!($B>ov|$v0{iDfsf}qgtI^KOM6DMH?8Q9+7ESj!DOG5 zj?!`x>^0(HG>RD&DI}}l@?v+SRKqpWxWEnU7`_m?Gd0D@R&mMVe~)dED>jJ-3mtPB zX%;^@Jou^b!?CDXq`D#~OkFkkN{=b&`50(Oioj_XZF-xPWi@Z1jK!8fIO%2&E~4`y$_e>2I&kr{(Il`l>qL%^)(7viW>dgxT5l|&7mR7zNXGR? zxfRFX=IhOmb?qBa-hEV-yg%CQCa6a<|4lg#6bp}a*`nOxgq-vbZ>KGyYfsTg@vC=* z|B~949{BMio6x6$h}>nLyM4u0xNk`0l4&?XYNonL+Hna={K{0(nS03CE{}F|?u=%= z(b|?h)a@WZnh02&S-M*4sg3}IuWyihW^!z_Ut}#SvIyBl4;v32OJ=0K;h|2nem7q{ zi6o!_Wz|;MtHE@l0O3#g%x@beda4Kpg2`gP&tDr5sBb8^8QStKk}!V4*DjAGpNatr z>pJLTS9VyunB7K~U{%5QbV`^^n*E`|k>g8oO81=fSHIDz(@w&-UM1$pPD@uR5?tKF zn;6U0%hQ4aF0p<7u8un?msCR&$KzQ7+y0y93Co3j3m`CcK5Tyb5&--8X{~snTJ{&_ zBEqswE=Ijwpiy%(R|~UZxjAT<4ifBUGaY`=)kOxeM;_xLydse~;ZSuK>=jw|`8Bm(H28c?j(#)smuOQ$ z&8~9}wd;~Oj@M*xx$S+BbIg~iH``Uf!qie*{K&qL-I&ciF;ZU0@KK#Xv(0S13KGhW z8$yYl?(iuTmXmMB-vyO*rn82BKTZ*J0!`WAR}$9+ zR({dFRtufKRd*N2S)1(nDj=WccmRv%!#Rs!JF=-_u_|LH35dvJN zI8#3makQE)l%Bav3-5y{Qr92}F zl_4SxclKn{o8j6|u8+6e{xTxaZlZO3lWl_6wXCf!6Sr@DN?fVjmBeAYR9VQoXPS9g zkQ{rfY4))zfWWab-*XJwn`yurgck;1UmndLJ+O?5`{ArH@%{drGOyh+a=Ed>w5%|Icfc~trvVc2s3q)ouuRLnn8Zi}sl_er$Q#?^= zwaUPZKU335K)d~+Iy5ICVPW12AU99?|GcUB|6lV&nLqul{t^`>6_ZpTy;Rra&o!}< M91QwN=6Udc0K2Me2><{9 literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/crowdfire/index.html b/content/ko/case-studies/crowdfire/index.html new file mode 100644 index 0000000000..227a5c0839 --- /dev/null +++ b/content/ko/case-studies/crowdfire/index.html @@ -0,0 +1,101 @@ +--- +title: Crowdfire Case Study + +case_study_styles: true +cid: caseStudies +css: /css/style_crowdfire.css +--- + +
+

CASE STUDY:
How to Keep Iterating a Fast-Growing App With a Cloud-Native Approach

+ +
+ +
+ Company  Crowdfire     Location  Mumbai, India     Industry  Social Media Software +
+ +
+
+
+
+

Challenge

+ Crowdfire helps content creators create their content anywhere on the Internet and publish it everywhere else in the right format. Since its launch in 2010, it has grown to 16 million users. The product began as a monolith app running on Google App Engine, and in 2015, the company began a transformation to microservices running on Amazon Web Services Elastic Beanstalk. "It was okay for our use cases initially, but as the number of services, development teams and scale increased, the deploy times, self-healing capabilities and resource utilization started to become problems for us," says Software Engineer Amanpreet Singh, who leads the infrastructure team for Crowdfire.
+

Solution

+ "We realized that we needed a more cloud-native approach to deal with these issues," says Singh. The team decided to implement a custom setup of Kubernetes based on Terraform and Ansible. +
+
+ +
+ +

Impact

+ "Kubernetes has helped us reduce the deployment time from 15 minutes to less than a minute," says Singh. "Due to Kubernetes’s self-healing nature, the operations team doesn’t need to do any manual intervention in case of a node or pod failure." Plus, he says, "Dev-Prod parity has improved since developers can experiment with options in dev/staging clusters, and when it’s finalized, they just commit the config changes in the respective code repositories. These changes automatically get replicated on the production cluster via CI/CD pipelines." +
+ +
+
+
+
+ "In the 15 months that we’ve been using Kubernetes, it has been amazing for us. It enabled us to iterate quickly, increase development speed, and continuously deliver new features and bug fixes to our users, while keeping our operational costs and infrastructure management overhead under control."

- Amanpreet Singh, Software Engineer at Crowdfire +
+
+
+
+

"If you build it, they will come."

+ For most content creators, only half of that movie quote may ring true. Sure, platforms like Wordpress, YouTube and Shopify have made it simple for almost anyone to start publishing new content online, but attracting an audience isn’t as easy. Crowdfire "helps users publish their content to all possible places where their audience exists," says Amanpreet Singh, a Software Engineer at the company based in Mumbai, India. Crowdfire has gained more than 16 million users—from bloggers and artists to makers and small businesses—since its launch in 2010.

+ With that kind of growth—and a high demand from users for new features and continuous improvements—the Crowdfire team struggled to keep up behind the scenes. In 2015, they moved their monolith Java application to Amazon Web Services Elastic Beanstalk and started breaking it down into microservices.

+ It was a good first step, but the team soon realized they needed to go further down the cloud-native path, which would lead them to Kubernetes. "It was okay for our use cases initially, but as the number of services and development teams increased and we scaled further, deploy times, self-healing capabilities and resource utilization started to become problematic," says Singh, who leads the infrastructure team at Crowdfire. "We realized that we needed a more cloud-native approach to deal with these issues."

+ As he looked around for solutions, Singh had a checklist of what Crowdfire needed. "We wanted to keep some things separate so they could be shipped independent of other things; this would help remove blockers and let different teams work at their own pace," he says. "We also make a lot of data-driven decisions, so shipping a feature and its iterations quickly was a must."

+ Kubernetes checked all the boxes and then some. "One of the best things was the built-in service discovery," he says. "When you have a bunch of microservices that need to call each other, having internal DNS readily available and service IPs and ports automatically set as environment variables help a lot." Plus, he adds, "Kubernetes’s opinionated approach made it easier to get started." + +
+
+
+
+ "We realized that we needed a more cloud-native approach to deal with these issues," says Singh. The team decided to implement a custom setup of Kubernetes based on Terraform and Ansible." +
+
+
+
+ There was another compelling business reason for the cloud-native approach. "In today’s world of ever-changing business requirements, using cloud native technology provides a variety of options to choose from—even the ability to run services in a hybrid cloud environment," says Singh. "Businesses can keep services in a region closest to the users, and thus benefit from high-availability and resiliency."

+ So in February 2016, Singh set up a test Kubernetes cluster using the kube-up scripts provided. "I explored the features and was able to deploy an application pretty easily," he says. "However, it seemed like a black box since I didn’t understand the components completely, and had no idea what the kube-up script did under the hood. So when it broke, it was hard to find the issue and fix it."

+ To get a better understanding, Singh dove into the internals of Kubernetes, reading the docs and even some of the code. And he looked to the Kubernetes community for more insight. "I used to stay up a little late every night (a lot of users were active only when it’s night here in India) and would try to answer questions on the Kubernetes community Slack from users who were getting started," he says. "I would also follow other conversations closely. I must admit I was able to avoid a lot of issues in our setup because I knew others had faced the same issues."

+ Based on the knowledge he gained, Singh decided to implement a custom setup of Kubernetes based on Terraform and Ansible. "I wrote Terraform to launch Kubernetes master and nodes (Auto Scaling Groups) and an Ansible playbook to install the required components," he says. (The company recently switched to using prebaked AMIs to make the node bringup faster, and is planning to change its networking layer.)

+ +
+
+
+
+ "Kubernetes helped us reduce the deployment time from 15 minutes to less than a minute. Due to Kubernetes’s self-healing nature, the operations team doesn’t need to do any manual intervention in case of a node or pod failure." +
+
+ +
+
+ First, the team migrated a few staging services from Elastic Beanstalk to the new Kubernetes staging cluster, and then set up a production cluster a month later to deploy some services. The results were convincing. "By the end of March 2016, we established that all the new services must be deployed on Kubernetes," says Singh. "Kubernetes helped us reduce the deployment time from 15 minutes to less than a minute. Due to Kubernetes’s self-healing nature, the operations team doesn’t need to do any manual intervention in case of a node or pod failure." On top of that, he says, "Dev-Prod parity has improved since developers can experiment with options in dev/staging clusters, and when it’s finalized, they just commit the config changes in the respective code repositories. These changes automatically get replicated on the production cluster via CI/CD pipelines. This brings more visibility into the changes being made, and keeping an audit trail."

+ Over the next six months, the team worked on migrating all the services from Elastic Beanstalk to Kubernetes, except for the few that were deprecated and would soon be terminated anyway. The services were moved one at a time, and their performance was monitored for two to three days each. Today, "We’re completely migrated and we run all new services on Kubernetes," says Singh.

+ The impact has been considerable: With Kubernetes, the company has experienced a 90% cost savings on Elastic Load Balancer, which is now only used for their public, user-facing services. Their EC2 operating expenses have been decreased by as much as 50%.

+ All 30 engineers at Crowdfire were onboarded at once. "I gave an internal talk where I shared the basic components and demoed the usage of kubectl," says Singh. "Everyone was excited and happy about using Kubernetes. Developers have more control and visibility into their applications running in production now. Most of all, they’re happy with the low deploy times and self-healing services."

+ And they’re much more productive, too. "Where we used to do about 5 deployments per day," says Singh, "now we’re doing 30+ production and 50+ staging deployments almost every day." + + +
+
+
+
+ The impact has been considerable: With Kubernetes, the company has experienced a 90% cost savings on Elastic Load Balancer, which is now only used for their public, user-facing services. Their EC2 operating expenses have been decreased by as much as 50%. + +
+
+
+
+ + Singh notes that almost all of the engineers interact with the staging cluster on a daily basis, and that has created a cultural change at Crowdfire. "Developers are more aware of the cloud infrastructure now," he says. "They’ve started following cloud best practices like better health checks, structured logs to stdout [standard output], and config via files or environment variables."

+ With Crowdfire’s commitment to Kubernetes, Singh is looking to expand the company’s cloud-native stack. The team already uses Prometheus for monitoring, and he says he is evaluating Linkerd and Envoy Proxy as a way to "get more metrics about request latencies and failures, and handle them better." Other CNCF projects, including OpenTracing and gRPC are also on his radar.

+ Singh has found that the cloud-native community is growing in India, too, particularly in Bangalore. "A lot of startups and new companies are starting to run their infrastructure on Kubernetes," he says.

+ And when people ask him about Crowdfire’s experience, he has this advice to offer: "Kubernetes is a great piece of technology, but it might not be right for you, especially if you have just one or two services or your app isn’t easy to run in a containerized environment," he says. "Assess your situation and the value that Kubernetes provides before going all in. If you do decide to use Kubernetes, make sure you understand the components that run under the hood and what role they play in smoothly running the cluster. Another thing to consider is if your apps are ‘Kubernetes-ready,’ meaning if they have proper health checks and handle termination signals to shut down gracefully."

+ And if your company fits that profile, go for it. Crowdfire clearly did—and is now reaping the benefits. "In the 15 months that we’ve been using Kubernetes, it has been amazing for us," says Singh. "It enabled us to iterate quickly, increase development speed and continuously deliver new features and bug fixes to our users, while keeping our operational costs and infrastructure management overhead under control." + + +
+
diff --git a/content/ko/case-studies/ebay/ebay_featured.png b/content/ko/case-studies/ebay/ebay_featured.png new file mode 100644 index 0000000000000000000000000000000000000000..4ad17a4af50365afac13a4b6f841f5ec07b56a9b GIT binary patch literal 22471 zcmeI4cUV)|xA!Bx2!j;q5PBz}Luk^QAWgbL2%#vD0HK2gq<54iO^P6hg(AI37Xv7u zATUxydXp~FZqS)AGe75Y@B7|A$@7G)to>bU?ep0sC!FUbLSIjVjD&#%0059_X{s7x zUb`^QR3ZY*Up?v`Q_Kr7Qq$Z603fA0ex3o`&!7VUr1_nU%}{2#7a=f&i!jsaTMa2byqF@n85LgBzA}$0Lfq-QoU=iRS4|W9-j3wu8XAdz{ zRsW+L=AAsdBMOCtfI!~f-ooBu!U%T@XBTK2Rh`L>LVERgfL* zuNb7KyYu(1*}*_?XSfU873G166ZvHv(h-3|csL^drQ~1D|5_L`Cc3(R)&6CFU0i-y z+5@HLg+cHK(qBe;82cjOAVatZ!qXiFSM$Og3dgUG9_3{Jb8!Ba>A2-z`wsVU`j@cd zmLFl?r^1gBlEWM;M8zErMIqdc5eR36AG1vVPlP}fmE&0@1?1I*!kk=>x$?<@{u%Uh z#DC_4t3pw5h2vNuu#}LPh_NUbA|Vcukifi!fWcTveklKuLl0S-dCI>>>3Igr;L|3_+71kCezz5J2d?thsYEBrrFUvhWC>~&CQEX{H6 z|F)>2X*zjeW~eXL%*D*#@4J~1-0kn)KO3B#zVAv%sJjRJc+JVP|GDV?+Q$Cc zrH*61uU!ZfcD$b{z>b#$+)fVkH}B7G{T2B~nA8930RG6275?uvydB}L|J{Q6k?OnZ z-;;PC>`~rOces)RW={OiR{M{A|hMX`E=E;yMu#w%`;OHjNYQQ(Pw^!WNw3!lrSeeTwTuMA(8;T-Y>Dv`=xJhzMJ7 ziVK^@iS{Y36A@txPH|z=IMF`Ebs{2c!6`0m8YkMPxK2cbEjY!6P2)uS6xWG}umz{M zuxXrVpW-?Z5w_qI7dDL(?NeMQBElA&;=-nJqJ4_%L`2wvQ(V|IPPG49TqHkl0ED|@ z?)vk_+};OwTL{A38VH1G8tMW7ep~>+tzZCP_XzVk4*+rhLV8R?kcrdo;JAISkF6r&oyr2k&%CyqiVf;^_N?}0~u_i#K zF^emSU4)dNuE+Nc^NZf&Sfr`6lcdXNvY7>HxoP1%k*s{* zY#v2zn{W2&D5ABzSU|66_|3iOh6#=iC0Wl3NV5HCBAo`Gp>b5CZ9Y*@%7EAz2V^CF zWw4kK*ac51h+Qv5NiAFzI6`_@oTQ33Wy%?L=-wdb_i$h{dhwr5oOnyLhl+NF)!ADyS z#B;{FEA@@%nTb&J8ND?oZRto7!F>;y{e2j8#lXhMYL7QxOuGD*3v)BqRJ_ncnXl&T z86`D|a6cte{WWbO0SaUNhn|`r!b)9cAR11EV!Xr#+IQ2)wZ-K_XsCm4T#a2eR00db zi)-S#U-a>}L4pD858~X#qpNQT>L|UW3l68~T=SJW6ELauiK#O4<-S}O>{8#i-pq;> ze}I?-{@^W<2YAHF8Tf89pEHt3h(g17t#-I0rK~U2PfKWCqPgdHm>b^?@4xa&7-U>Q z1gtRZM72Km7My&%)9bdX;$bI7Uu-$A^XP(rm3_1M+n@-E#i5iMPfJ$BM`Qa3#Ri}B z*&VLN=Dy^v-j=Qm#1pGw_W$bTJVnuqZ-ejlkwJUCUdDrb7O#!XHbFBl>?xPo!80Pl zXB;%>oEk!3uk|p;bL>|NGJAqOoS7OUwt}V3&65nm71P;s(_9sLS~Z$pDnr!R79QSx zM~rGKm|o4JCWH1z`{iYggELw!&=V6tC|#DfKUXt8mu6tr_V!-tUUqitk}SZ0$lwlt z^19lzbCv)x567*=%0htUvw{8BQjP)h0&UNl62}z3?of|y`jvF)jeepuZk(XTgC3e* zV%FJxM;ljq-67LmNhzP$4>cy)F@26miH4dtb9^!eK5j&IXs40cWJbS{zREl^c>Nl6 zi|cH6rhQHkp!|8Z^PB8==0i!lGr1CT(kW;e3MH+>`{B1d!##HwNhSn`*H{x+8gY}a zXg%Maa!miS)Q$VjlJF6MJTP`RgzBQ?j+m7%nLu&M-P9Bo-FuGFI@Pl;Z%&m?9jOA( z|`{(c~4nCC!zap0bd8*Cu#MfjVigp?ETL?M?~Shv+qdGEtR@4Sbx=+ zIot=i^$--BsRj|Ha8`Sf;%&Ya{WRpt-DpcS`7q@Hf!_uL+?`~|l|$(kKMmVi5!h74 z&3$mVe!ziAhKD2966>X}hrym@l|2qsOz*C-F%D6uA?Ou6_M~N55&DAsnnJ>!cn@zi z=7c9bhJ~uzit-4G3qDO#TF}o?R#u}R?FyRC(wL=BuX1jvP`#vdP(Gcip)9*c(RKIc zMX!C=d!+Hwuk-Rt9AtmZbW>SX~k1*}^-;lKeNJX{)eX_kq1DXkw%d|QqZGy*W9|n4Da7(*?qvL7WbT-#xNP#6 z;mf5U=unQN^-&*_EK8A5B>rWSaLP@qj;fW^TtmbC75}g~0lvhF$B`s%PQj>g+UH?S zQEj-=%s^=?3yo{f*erHLjF+`bmVvzY_-$NBXU|d^1JgKVznYh|Dgs~$ylo>CmgASK z9Wr!sQ(@Y7i{-AKZ)F)V5AKXT!hP&F{>pLLlWxN>SM!xJr2X#HQComRv}TIIor(c& z#?Z0X$&;hJ__%cGZ0vZ=AqlTC#>XETR-2@6@2nK8?Si1R_vdop)gj!y4BA!o!3!E= z8qb~Ni>s2uItIcEKv!d~zR0ja(exk!vbR-a^|kW|!!&o8ZC09&KqS}IiLcfX2QCQL zeJ0BlERAM$hNdT9Y#tffLTns%AjB1>C--grMmb(r{5qvc zzH8Gg%=R+XPFMvuW#hLe#G#?=3ftQ^l$SMKi0H~xvN>coJ@wz$lVv`X9%481Mh zm*e*|=s97a_VNS0gDQ7*4;E1Z9-UpM6BbK|Owx^!@l(a7fJJ z^No$&PC^3BY~SRL!Xpw#C8k`Q_KkEwaM$|#TCl!uOaWVYag2N&nMd`nP!}&T6-@T1ZH20${ z9pb@#0>9x=Aa5&u)b;cuv%GV;5=|=7eq!;;$%zpyxpdq%9#9F1k2<5~g30>$c*QSU z!xYht^L6^U8~2x(t|i1el_F;J3TVkGHP1Z3x!-m6I^o#s+Ga)~)VbNSv<$OPICF8k z-<>5LKfIY*N^D$%W46?S(>$kr?dDgvpeWO+Ud(Sf5YVW#A*vHU$&TM_x_C-h>Fke{B_+@C;W~NL&|?YQq9z`Hid6p2q*O?mf9yy zD3>-MBAFGCrQ-?XxNgy=DQX`V`bti&r`-1drAJz9M0`n zBHl8DpX_4jSnunF#r2^hKDGMr4vWs)Y5u9#nb}CRfy%OvJq+HMB%4a+akj~zLtgFD zYpK38?=E%g58PwZ*Q`-|L`7ywZY>Ud(ER34JS}*d>nSL#EA0$tC zT~>QI!IO>09B;1`l$g`Ip+TFJ^8tO?185xX@)E|_YIl#Mk_TOs!Bj*XJ*88`Ze7^|ST2M?-m4{?LGfW}BnKk_0AQ&)RoqaH{ z^Bl3DSE8ez`-VNzF)MSdw|SnhR9MA3#IexBs&t)0aJ7acEU=4Nj8&`G?78qrqoFtWgcrFCpI&)6Rnx?^xL(jQy%l%eHB3-YK_Fc@l7hH)E5;CWR;c9=&S}>= z(X*uJMo7rL$|fdaU}jq>(c9SN|3P|RI28R%DI-Hf*D$yAycg2}JJ0>Ph*p*oiTtOn zks~H1JB*YLs&gigLb2i87er`Bic~nacOU-<^)lm1~S*T529@iDi%)S^;Bx@(9PR!ykYLV0jSu9rTe=*nTmhm zCW#)Pn*~G(>I%-*eK3pS#jh=^I5Te*+I3#DjDX|)3+F3a=Q^o20Nd=Kgt5(LA6x1a!`#iD#uY%Sq4 z99@Yws`ZMypw6Gmx0cI5)o+fs{AdeQ%R1{@s-mZb$jV-Nv{G1GPgivw2On~wg?IHT zeRHKH%6k?i4%>iO-A(Ux*>;$C9Cbqf_)Go!lX zoc@PQsf~m%N)l+=@E~0s&&otuj<-R?HxY+$_72YvzDv)FyveBuX`;I{UD739NR`{$ zFeV7!%`^dny55W(^_;C#$OeVAUj~wwN{+)->jfrXQhw~;*+YBxMJ&*Sn%(o7#_mIFB-foBWt{nfNOGDBJTwp|z@6&hWmMK7 zYie32MLG`TC;-+hU4kBerh+s(pRONO)NQ+!;7!czo0E4Q#HOdQJ0#&EO8DM~D?Pa> zeQozjMHzR}qc~n(vC*+kj{(K}rixf)eSanKK5daPpV;8?xcHhoiNzw>-G;GyW{N(z z(~flhx6d!qs(Yi&HOm7Oiqe(MNZZcU8n-Z%-?K2Q`nqqnvwuHdHfOGO*WyrS`*X=a z{~&~y>KxJB*WLEg{f!My#f^Apg|{gVnjhO;ncHoNId6sRx%nSOFTK5@S(^L?Difw7 zrcYlisr_NcBEOm-BFqfJbpGB2TBUa0KIy)2VJM@&R`wp#(^9GNm`~~6sij=)#{44gfTXxERE)sEWh|Gk}Jup2GHDtoSa@&WFW)sB$SD;7>yXUH-hi4qU;?H5f zFI+aP;h|M))EZ?#_p&aMo#lTH?yzL5q=;p|f}h0@~wis;9t6pvc^g##YY zywM%+d>VoHWTpK`prVW*sk$Sy*;_xxq@F$Z)viDUdNyYB%Y_@Ww9V$EPTJK0sh?z@ zwlet7Yyd$otSOvKOwy@YAT7$}D*F1GfXvW`SEW8`ZuKj7*?yBUsEU4Lr&;w@T^Xg% z<;27(voG~1hX)CzSDh+5YsDy!5}NF|l>6;!oS6CHQRX*K>m|?2XYqJ~^DKY2Osw~s zFL3IBT&>$Tl@{Ft*L5_;dg52$wbh0(YaAsfygoljHum<33-xkISGkd*U89Fwb|0l~ zo_zZtO;wrKmnV)inNwy_<2*8*%V!Cp;R~y{1~owK$!IoN5fw?1-7?l*^ZjcbHT9Nu z>+PJ{NM`hA#GNAi^dU8nj5LV`H`O+a3sndSF|~B_NIZB{qi2QtT@(q?(<>UUQ>dOE zNcWd>HIdJj4FPS+onGDNMOCSDTvD9gRSbZM@rT?P+(hItklnQtqR64RCnLH?r?D7k zdnGcQ`)wp!hG1HfE?AxRvt>0k-GRoKF}omp}*tee4x~Q!-dAxTOQ#$Z>KM7zTwsv$(VYX1Ms>5dMa^-((_Gy#!`&@O~o$gS!V`cR>tIy zn)XJ~TKbL^k;94}AgY+cWb(-6?2%s7hD^}Pil11T?ZB71>PM>!^&tl&S}%HGT7s0J zcNT@Y%_Ps7lXL*vQArG~0S!{7i?`Sv7E zbKKdp?;URi8hNk%zUjq;tT?0#b6{;rtzc9$#OtIHaD`R7%M7NGx~ng?B<5SD%w0zF zHEp=;GKLD$t)N!>CQN-+(KjH{CacJDyt-KQoEwyIe-ofo)zXE&|4(_gFig&!fXe$!Ta^+PgkbqoRCncBf98PSe+ ztO1cT%j5Djt8N|MMK3Kz#%u1TDefEf2Wt=~#{u2$EOXvLhw**fR+wYtzFs+BVWe8q zp2#^q@I2N~P0WpbS~2h&?@hlw&gFYz1y|WAu1!wZz0Ur4Bk0mir;L!$oDYmRvD?nt z))Jo&R_CTXX>UIy)J`{0l3-Ok!^uB66Bp0?Rn%*{*;VRdlbHai0o!)lZ%&Gbd#O~v z0gJ0Qkx{cf^YxmzFU>lUaH_}GOmkIVIbThG8$&4xq7;#A|8OTE3vF)l@#2FZQ!8qj zrX8)h*h*7=X z-;tHdE{nWlH%nDfIUXnDhI2(|>RwB4_l0Er`soJh8}rJ@*9wQwSbb6V2wVj1hjR@c@&YWG?&iopInTk&^0EA7POuQp>|6X0 zD$PQ%k+i9Uq2G|BFBkM@Fx^jyKDk0$ZmPjvBAs5&st|4VADpLtEHY# zS`XdQkP+Vi8}&F8oe!-`kN+smc^~tuLreQ}XKO^Q9gxc}2>x`SuECP3k zGh$Xr^TOrL{cmA>H@|$OnALD_m=R+jQ)!=}*Sc@f&p8$P1lP*jH{@vJg;p&`zF zPE(XH|53k1E?O}}jJbP_mCY??zsu{4tE-~<-m1aCZ@I6p=TtEC!Ji8`JFutZAD;Ev z=H(>#U9cJ6q+30yXrt%cd2t12pt%OjiYT4$_e~qH@{INrd~DTR+w39^>D27OxgEK2 zUuFH?^UR*|qHR54W*^xE;YN0^A&{olI>T6QC;{*IH{pDJVBn@mVxiYv7G|_qxHyOJ ziVVa*tE+R;^TJM%gS6YLPry7!a452!q~%G9ksta!_0u{63At47w>WB#;hM7NX`GNj zT1q))gO7ce)zoer1*7TcBR(8{k-JO93>jc1zlf54^x2MsZEi*?=K)R#nu+^%*6sMm z&z?MAUgt-Q8@^@*l%jmllNUo5hP5lEjqK#JTlMydM(_8Q+tOSpyiGqE;U=5HeeQf= zh1FP-I*tHU7tL9v{Pc%|`<_^+UyUk@NIb>Q6BLuxXRXV|Azu3~#vW zonw3}nW(u>dvWu;#)s;q^RN3}3D{442}-gnjW@*&G(PMheBM+YGx$+4$8)wJRVA#{ zV=Lgc1!+f?TsiP_NW)w}_rqr?$OEo6#0}<$RSop-$Fp~Q6R8CrWWo*>aH^~nH6P#` z#B(S2O~#W6kVs!QQ>0vD9T#f@IrDEk30st_EmZU1ZcW=aOITFs79KzE#=D-mpEDVr zKCVA~F!YYD2;Z(Fll@hHf7>wLdw+?^4g=?-oeiyt8VBGa>N=(Oy7H$&*~>#BhQr;5MgAh( zD)WGIj@$!Qbu&~Y%sz}k9krV@0ud-kCV_tAgU0K#Kr~V-GChmU_Y-~h8gchKL04Cn zQL9&l*9Exp4b^|!&H-K3t$;H7wQC+`OkBvW{9t{51o;Nka_M8DTVVhrteiehZuCiC=Kx*#TDh(*9`(qIogv157={!#GSF)CeEfcqik+^tk-!B- zbk+fS5fI6b)`j7Hw{&->{^-Enyzrytx-3%mSjXF?yPavN<1vZx`f7IOL8Z*eJ663mPAc9E9~vR9 z3^xpX^wb{5+l#jL!mMqddd8cj7*M{wI5Q^@TL%ApU_3Q#H&}n|X+%of#y)TED7MC6u zoHwr(rTrXR*SP`G7$~%4iF@|*V(l+3nEt}`Q~&>SargN8@UJf!|5@_Hi@R?wO&|RR ZFa>NbRFW0b9{<#?mYSaGQ)Qc={{mwMEo}e* literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/ebay/ebay_logo.png b/content/ko/case-studies/ebay/ebay_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..830913c52b13fb4e808cbc61bb03be9ef990a724 GIT binary patch literal 7316 zcmdT}^;;B z2HI{nn$Si6KQ(R?9RdqbDs-JE37X%TNv428K=s~uz2ZxYMV%4Ks9oa8gD!c zSh$(4@-?WU`wY_gX&4600tNxfe&nM8is_P}rXPl15EC^k7!ng3fKOc&IXN`jHfNvL z?3e7>0Ak}_l&G8}g_R*vE|5mnNF`?^SBv%ocv_A% zUzO@}Y9Zq=g2dCZ;K52o`;3>_8mtgS*0j=l?LmjrXGo^b6Q9OC@DLYq5??7*_tc0I z!R9)uy?93Q-d&Cx1e6X1N?R^jw8$yhPZSj4vrj-Nv*QvYR zlqz#tB*wqYy1zLQ?NyP$G)C2jp5HG((_60J`f znDldVR4}7UU0fNMn?s8`8Uizb8~Q(}eMN59fL_!+Iat&^UT2#PJ7w93gvJ7dZ0|&S zvAM~#$a6q%JVPBg^@Hz$2Jc$wj>^u55dz1{*84XDJREk|EQ(+YuIx8oU^M)^Obf4e zcvEy5y{w6q7;^$Nx;2A$yu4Tg(~B-Bm|Kx~`2ue@3py!WQ_vZFe5^H~sqKT`QEO@` zp&;^~3OMnweY!4O8>a`|K~kJF>&TgK(XvaROEBa1qGFN!-Fr?g!PE;hNC%RI3Mu?z zd7A%MtZw^NqChkLFUf1Ei9e6hof(Q+DZwVNRE(qUT8PGT_t|~yZEt*T4l_4c=4-^v z2Vo&gj$m6*MORnUah|mjYm%qz@ic0}wY&c#47WF;yT6xP^&m&S>S7ot_PfVtxl^*~ zcj5F7)9QqrP7iEOT}ZvVI`5U^>sv15{oxdHWZthkXFY+vPs-;%de8Jr-o9gpVXkng zUaoMNkvi(;<0?sCk;#otJLp#-Xz-TRY<8DIRseuy<;uVlOYkJTl25yTO%j-TVl3K0 z-h5u~>Hk1_SHaBjGeq`$5!^-hKwyeti|MBs^MypP{T?Y!pb#LW{9^LFU)KCVlr1YM zP$Wsg0Kdv~{+7f{3h^o;kzdYn%So{djfgni3=r*dRDm1mM?*JQ>yRJ8a%%3VlLz$d7 zlLr3WXZhpma-#(OUj;e1#e}y9$hcCxNKugDb3dbX@pDCKaq`4-k>IINQWE|~U2RL3 zh&|ravg#8XUH{?lu;>(t+P%9;;?s%#b)8bZ@)(Wyqk?2%_fjB6@1&W>8wSeuy5nJU z*4Wf62@g~W+cLBy;`EZcFp~Yb6^YrC3uY)Tx|7yAmkkUAW2Z4WA6?lK1-QfNw|ism z5s7xlD~9r{B!`vPFi-f$jK5n}LqQ!!gE_ij=&(~0%-~+%K0>p0ZZCQV|x`b`N( z1ZoM;mT$^18yDDk`SY6PutzDKTB-ElbRi*`|GGxPA9sX`J6lqSohz(;t?)7?Af3m? zxnC*N*~mbS5B5yps2R z{3y=QIKO1gxj-)mQ2o3;8puqD^(E)5HoQBPUndpl3!f*JntrzTS7}hWih2+rRm~#u zEuiYEL)#qbZIn&S+ejheY}+M-!{f|@CJ{ZY=cpC;-UchD=yS37Sh zT&Q<#b$RGCIjHOVo?O}PBb_Dx#+Bx!0Dx;L#^joMc)V7)g_33K<^1zAXPnq&@8w>z z{_hhI#w(yklqYVau&>c&L+@DA=qPzetiG9;0~Yp>}UUG-mVBx%aJD_tnj92EVj}kQTTT-<>QyL z;m4|!&8`D!2WwKPAtc%7)4&`nm5Zn&HlLqA8%oTecND!R+-qJS?;$-R;7|rjhC_6& zG0h!L&a^{R*5*HWy=!5^<)H(xE^Z&4IK zY&=yItJi+D_<^nkl$K4yjg}Y|YuuF`?lFgUiD1||0eU1ep>pFj^2$`NZCMo*o^l{Q zI_lBaiq;BARp(t06u*%gQGWjT@r~H|$WS<~*14^J;zO>PP#jld1m`Qqx$`wGecJ;~ z#(%k%rSl>mp=FFqo_7v9F!5fOQ|3pu<@0@oqJ`~|NDkO)FhAVLO}Lq0(51kFaBA3N z);`5ha8AQH)X6Z46O{%oL0m61A1nu}idx2yht!bzBkz7C75??Vf#gcHA|rN>hvx_` z<>d=sSw8+X=%x+5v zn79uWAWm^ZgPEJbKQXM>?f~l}xp}vf(A(ul-3xu$#G{=lmqcNtN72uf$Fwi=!FKN9 zl#PL{F==HFljYT@6Dj)Y&6mM@z1p7R&z|d*7vrwt?Pj!KfO`|N1Y(g8Fz#NAdu9N0 zw~zS=!>36VpLYbr*Hk(5*48Dm;hQTQ5tNp4|8X3LVD=#0R3w)lrcWD2PmL>mbQ&5P$B*z5v<jAY45fApVxIc4Wda$i{{BQTQZR>S2lFS&tD@e%F4XW^`guU~ zt@uu1&1b#l=d*0zif)=jk#%gXaeDIAZ6Vw|>(RPBnoAy(dft$=_ByY$w0Enz8FYL( zc+bUF;arUeIw=GoG8`LQQ%X-j7@mRzuBM@OgqF~tecL0G<6fXg_KSIl$iT>)TgKM# zCX%0e$O!9ApcLF&!-Hr7Q~{U2BV4YyUvJ6P;81P_Gbk#8$#B_@|^nm4JxOZab{P7&Lpg=Z+BsKAaL3ys~&BL0WIwyAZZ z%1=$eGQpgg$~Ks^QUo_kHg|+X$MHF?J&omHE-d>AakK)(uKlD7^q@+z2zC$)0!;kH;;1w$y{ix$lovAB6h5l|q1qxv-t!im9~4;b zXHdBXWuUetQX-56%)Ux9Xm}FP6CJxvb=;~)dda~VqEF#yPE2aGyR=Eb;oVl@P+ z@g!&<;H+x3S5A2-7_HFDXyKQQJPuAy;fPl#n!H421^Ot6W&`8d^!kVbg>8DiEo%B8AEZhFzS;*(H6%Q+YHIve}X}R4He(rGE6;$_sxEfDoV!T zi_l_%hzjP?J8}&LpIpKUj5(hfMm;qguBOvBA+;*M>GnXuf>GG~6aH2QiB;EMWto8A)H z?}F1tZH(oLF!YPJfTSP%eU{#YW%tu2PrGp1UwayW%h7MFFhV&y4ifo$SBxrca^g!B z1;lU9fT;jQ{SBGdejWLiX!*hQ^${XJanxZPh=$v}wCd|cb>q`0+tFjv1>py_X)A5k z*LjC`bpOf#RKFp7?6z{`V`^zlPp@>~z<(L5Q*lxs%EsU(JEv(A;lb~+=8H8LugTgS z@#N2EFkJO>%(!p#$X91Em|3wr&*=*=59nwp>k;IwU^dufp{87e<9yA`njJ2MXqTGG zXw5S%x59K3{qPzv4e#|l;pW+ZcK1PJrp1|lU3VeF#+*?%j3|A2&f1U#xeKhp=s?zl zCuE9ZWAv{>Q!nmD$lbSTd&)3dBNq~$1;|&L{V&zCMyr)S>02@%sJhrsnhn_V<0uWj zAB%&hDyy^h;(Zi^;-!!pajW<5cjA`dnSG*oqo1B;uYRuPhjrMP!g8}lHsMz&1l(gQ z$#B!;tjw5WSdoDK8-_y)qx5OL9MgUbwDt9i62a61i(yWG#kzMk+Gx0C|F^Y%(k!lr z1dSiIN}!|#c2;}fR z(j4T8&z7?qfN`Iw7V>{Os-&XCu8vV(knkogE$YuQ&4icx7HR|K^q?JEu*G1`uP$5O zz#FeyAiAp>vJspsWOHM_)}uqE|Ipnm9ek_T`-zz?_zrV*J~=_w7GPM+mw~G64oKG9 zZu=l8FEzf}kdf0$M{~GlVH?VtxVys+QCg(72>19*XzZUhLQxIU-1Hd){1=n4N@^c0 z=`>-C_8D`y9$0p~H~XtJX=yV0tMHu?RvH2t-5u~baHrc?N<=nV0{VO~tT)in%_0ek zw>XQsv1|{+yhFF?a?Q_)MyQJJuoiOOUhKpF&?{weLVf{M;yq``XOC{VF#)SszsL(p zWHM<5LMaAT)KZUD9RNf1ik4F zc&T2b07JXw>7u))Zy{eWA)q!cCcC3qeN?L@Qbe~wqMz_nxQcke7W}=no=7z9hiWCQ z?~RBfDgw-dae=4YjHy@}AymWqk{7dZ$$VSvjksz&y3r1lG0TMkqymV zBt3*KHc@UkE4b;r=o|B6T|EX)*9F!5gfL=$IWme8W}pH?chv3gRyxoOaF^cS3%lY# z9>1}fw%WU+VrJXXRaX^h(@&Hx2CgVw>YWxFp#(z{RTbOSG3GVc-mz@#$atXM9#Y7Q zcDD1~lfxT}M$yYJpWV(IZGw%@0pnow_lq%An*)%BOw}sF~09BA9ky%AI|1WOr*ct43_vam;B8s zhFz@V#9D;#<>V--i#JMkk~ewqYxMYv=Ptc<`tF%?MOxH%aH)4(D#&QFU%C~ZdP(p~ zxvxv!d?%$dIJT`wKrusdeztMblA0oLLiL#izH|TyXUCuFQ-Y01xnusnkw=O`{ua!6 zv6Yk0<<-BQ3qonMTcbtyopuN$(n)tJQX(Bn4(z0TcAwM`0Asqf<+Q{@Rx+`kavKJe(IRZ-ZH90AX}x5hIdaidVhBjkgh zHrnr%{+}W>ecr=W1KYh?P>Y9qk@&r9_P5q=I}FpEHjQP^#Aq9R*w^Dq=~4~~oc2v1 zN9E9nVVe;$=gaH6lk-9%fA-PUd;i~Ignu5XI48l@oGS)Vi)W<67^up8K3UAqR7NaG z`nGx;aI%7*pwk<+;J?t>v|>d^+&qt%@1y8vohQsc{0Wne(VJ_=sihpE!;YhJNEnC?`z4v7e00$&sXEi(t7I&e8G-Sb0M;g)>eA;gr`o+W#S48!t>yk| zZ8UtjCjc%YY}y;StxTlzP2w*73A3DZC!YUyHJgdBzN<|c;G$NLP741?Swggtp-(3B zD6@fJJQ;6NBTxRrLAM$+M}X-MNg1%mOM;Ln-K+y)#J(-`*b1;&sv| zkN$XKMzI#DXHo7#@okr3PT(ZSvR+>feY|3=cd|YymNAez)>iqe-7OelH%IAXsU?W1 z01?Qb{M_=9c1GE5^p0ljy4n%LI zjfR2D#?_fRX~eymE2W8QVvi{ycD|YHq%w(~WPwO4Le6li#(j4w%XynqXL4%JkHh1m zsT=BPG|~8}aF5}-brWS@^&39-j!dOD_*v^LoZPVzPn9S=uTB4(`b=x_``tyNB5n`B zS=#!521iSo#)ijec8;++3N(c#|IMbVQ;`_Q>cEL}&8Md=+E`ERe9ba`+9SpK?0>gr zsWU6U$2#S6fSo|NK@I(Prs}iWQl3ucZgJ#BpptUox)5EPUQ{5x+Xm}hb|9f!J?*bhY_a+tsUW9a$PNjW{-CHl?e%6e zub^&;R_MV=fEV(VqDW`#Ex*zT>Z~nCR%cwk1^-YqZ9s57G%?bQ*pO8C0@lje-Fkr!4zvKGBrvF$$Z@%`+m-9V~H%ugc#Mk z|Bfpl01;XhP9HOGqYqN~xff0sxgPzR4()+wu!~6(@JAFzR^}V}9e5TOp~11g&$>8W zk6vD-�@ne$Iyo5*T$=#zY*HUkY0U1EH1-m?tWZB6`&OIm=puZo9A4P@>qpIop{+ zuYZk~D;IMd4A9Tg*v9H4b!LxB)n6Xe3)k6wvB<_N*BxGEOL0w9bw#XB#H`^g>rgPdhmE`wtd_%1M{XD0~6ByZ6(F@U^xroy}(PxVH^=;r;%E h{Qu8iUnh@PD?dDlYTMPt{(T#>P5!b+nYph#80h03cIULFr-N zhq2E<2pIdiq4UfF`$puVV(bn8B-B5jARsN{3;>Ao+8G#Oj5IaGEuC-hT39(>NAr5$ zaKWkpKvKrr#lq47je%W9+t@it!B-nw;V?TZDfne!O+HN*1+=Z5imw}5-&f1P($~RK z%nB|eO)Tjxjty`Fjj@1v-*9wt7x$Ke|B5S)wSN{P;ILm3jDr+h?q@=nk){qz!PyNB z6Xq4+vE&yNfFZ?r`Gv&kYjHi4 z((mc8e@Ve@F&Gzd1j5V9i`Pq#*V)Ym>n&O3(elE7o7-x4|=l_uMkLv%J7~3bBnt$Z}$M)X1 z@sFw9F^Zm81ivHwM^ASH9~U%25AE*k;bw_e^u(SD%Rikx#?JcB&iPBEpEdun?Pzbi z|KRqs<~O%rL*cg`lEj{?xPlwn0^{sv;Oy)u{o5$h`4>W%g2K;HCCn=ft zWQi*{J36~zn}wDZltlbfF^<=-(I&3yGlw#Il_xS{22uw&wXI^Tbb{8!h%r;tSa{1kAs`)gUxwwgA#{;%)7Qgy|UlQ?eWq)S=O>gwGUmZPu>5&3_*aw3D=j-pTxYd7m zHU96exYd7m{nPW3lO0A{;P-5QE5`BqxdQ*TH0WbD3utK(VPQdGNyOia|L&n<=Z$tW zLfK*Gu=~%EDZu|Pr@zR_6e?I z5#b6>aN*K8);_^?EFxUN2`*e3$J!^jjzxqkIKhQW<5>Fy*RhCj1t++0X&h^x;5rr& zuHXb0E{$XD6I{n4!WEq0!liMneS+&)M7V+zT(~rjwNG#ziwIY6f(w_%vGxhBV-evB zPH^GUIMzPFbu1!W!3i#08pqlvxQ<1HD>%W0OXFDk1lO^Ma0Ms0aA_QCpWr$c5w73_ z7cPxs?Gs$bBEl7%;KHSGtbKy(SVXvj6I{47j#J^)xU z2LSOD0HAY@Giz4@fb(9eC^-Y~(Pe+*n2ceM39;?Nz>2|Y%Tst@K>$JtfYeOhD6+D} z#iB;k5_9|2N;y1VsWx0JGC=!(x17*ROg3N((9ei??P)v_L7r#(s7*1KqZ|g(!IxXU z1v*vzeRJJ;dv5x0d&2$Z6k!2X!?=fYrID=m_N3I0R_WJXE$-wORKe4?H|l*+KiDDc zS$r8TVB(iV9Hi_SK`(xpKg+P@i=SoxxjMj(5Q<0qa_6cvibm&*mMG}XG(JrzwU(@f zXrPUj5JPK6;NeE~I4dDO3;<;T_(4QH-TSJa?&lWz<#(Sa0>X4)oU(xI0R#Y3H72yg z1Uzm6Q&NJ#NzQ8cVe=c^l6X+yx#`#e4Twk%;FND$*ZZ!(@}W;zR(5$74#+-&AH9?j zrPmSVwkgQ;ddma~V^ZsawN2Df-8;l%lUHX-K(NCp=cSb8<@?Ry1hD0;o;#b-g-zo2 zG4!&rU>eye=K3EO4oC@Lue9fwYXUDEQGfALL;;E{*GZJySV+1pPFbx`0+bMZP>{}H z=KWN+A9x%+1Pipp)XSb7)&#m5+?rfv#UE~mr>4qKq2VVCGtJsmAP?4GW_Vh zEtmR&>t4e%Kg|#xrX+=48hZD;VL{-xCY*}702Cw#Q-Xq_@TKG|+1a=U`m8|S08tpY zG*OsOPR~G|geC|*H=5;w$z8s7{=MY$c&s(|h&Sd);&sLAFU^A8mIZ4KPH zwJ}R)SjU*SIFZK*`nIOFx7`|O3Xu)o-C=->z*s^`bqbR5TQ{eqQEMH_B^3{wgE?N2 z;sw#Da<;C$c`g-S8|k_JB7NBKedAEXyMeS?y~Wb(n{+n}bkFSTg#TENB*@`U+})tu z^Mr~A_LQH~5k)1%plcy^hGW8m`vqpARbEwVod`4E`CId9tTOe9`@7WY@hiDa6_AKR z4iyM0*tndMKd#VKWGI4Ski!K1ksVXrtN?(i9w=@?KdQbGA%pN63Tj+n57oF?F`E~G8w)s%2V{}oFx zF|*I&3;7N9S0u&5vy;9HU5BZa+laDOx9>_cM8V%?8i%7-oFj)>z*J$m9Ef-0BxUjH zgJ27^pwIS;m4GVtHDKt(Gj)ctD)!pzA^JU^NJ`R zQVB@D{LwgnDH6Lx4LN)=Z1p}0L1nAV6Zo%X>{Ah^`;ctj@GR=1Jw8Q19wIfZ@EXN8unZ$_Q}rgEN}anZRS)wvZyvfm91L%-ob=m;045DX0UIxNtsc2uw_94f`6=+|?x#uVz{V?yEVo)? zuiKXjzi|^DD3$cKq5xy}r=(f$5KH!dO8!TAJJQ^92+3#lOVwA%7-hC%u3$F@cmXO~5+-L(WO1E;}pLxIT%O3Tu zNPlCn%-)a#evt+wEBEb1U-|w37oC@{F;jTKYC0WQDvdMCC538EAW>7GR`erNMd#?X z?{7mWPlGtYS!JHPj|PY1xt$GX3sygp0E`m;p#D-t34d^qS%XdA^JYt*Y;HPb+1z&z zyGJcMw}EF%o)uSO$8(LG+3~b=k9;iIH3cu-%ehYjl-BP(2r#VZVtoxVZ-NeJLI zsj=F0e2HyPX@+-xIenmT@L|as(Q`?dN(~C!AL81wPoSIQk9y8RaVe_w<&%aYUnU|z zhak`RJf;1mB4WJ=NLE)-W|5_r<{Ryy+7%%m+w2M#Ki8Qz^JHJw-9j3zuhgpH19s)a z1hKh7m)ZEGQcB`zcvWB_D)93o7^AF0)Q^t4^`j4Jl4)(q9VNXneVFQEA||-JP^oAF z5ivOFxw&!hfboaLT=qvV>8^y7F{oRl%xQ4Kci*fv+>L%i9IO|SSUa)m|7pyqS+7DR z-i`|4|EAU+ETy1;FIc7)-W5`#%-xe(RD9m@-sYpwNX}GiAyPJw3g>%?Zmv&G4^7f` zS6aPt$NTo|bSo5h-N&F;+qlbT#g*^yF`dm1-3T4M!V#hZ0|Ji9z8nZ0-PLBI1#g(B zf)XJ-dIwa5s7ib#$d^pzSRfP$yd^+34chbvez-K}vFx2z(i~s=iCG4-G_|=s(X#sh zS#j3%OnOLh@^0L=nu57QN|GV1GMhSc?30)h>Zb>#9Irld>I}MLzP_@S;o6Kez<3&jY!IY9vi7T1+~-eWZ-bMv z8hjgPr_&)IryE-@sD*lwm%1!6M!Fu*px?)v)iVWx1uY7v3P7AcfzSESiqyhR@ zKG{?_LW0Lw)QN_jR~<2WKp}(`&goKWIzn)o?g6UxaOQ`@Bet~G2dnD=bJsp2?zM;h@q)9LfEaL%NT+SN?2;OP z+`C?bC2si=Iv6bn<;U^O1tmh@LYk{o5_?iUd-`c0CM+!Ll~GP^n>v4X%agjrBt%=# zi|HAX$cfuo46o@XuLf+-UP6i1@tKCB8+*{$BzEF*^&r!^{#t zw%93FGSBYCCnH+0s~al;oYiF*bj381H*9CLyRx5Hw%=;Anlnp-1a|VT8~nU2gK8 zTKl2J-IWJJJpoL~y?Vr@?NpR+@4qbtK#Fn@J_+5D&h=4WLLx>FBK(6I4jPs4Eu);E zCY@#j)!SE?vSEt-+QhPj?RrW^G{u=5;4Wg}f%|m%X^Q3Lr7J4uiz{_d2e}*&czA6D z>NdCjfd)~V?lP$) z1wnKnIuV|`sqe2pk|lT?Z-r(gg#(!Q462nUP|%D@&CMO_oB9y!C%nN!tHK9Wy^p+} zrfTiqk%fM}l5HUx~Uu3;|gay=DpiyWLNcmpOD`7ZvqphBGB=gplvm*PUB^*JjGjCV)W% z_PTYMb<~uya%%TF`*R3EG>TzWA^8^SK~Ra1>;{6AAJHmY6xUu7KQk9UJM$vj-2Snn z06fHAQF{A_UNZwv#a5m82&P12;(nvuT629e!S&mP03~PNrN&H_Fwqx6`V!6DmtR4a zKu}-1%E7*imE1HQH+`}p5Ze4DlEKJZ8)YWC^~qd{&ABRqd8!KdT_i(ats}A+)M@fm z-VBJOxd|0kb@PoFrao`sX8B=#p#6x|^-gT@R4TE;!;+nmg?$V)@O{;8LRi1G<^2O& z!{XScy3v-p_eF-Rl=)&q*LN1zc18^I3}efb=3hj9cQ`j;5RogZ*srB$-pR@d$Ypy9B$$wi9m?xF-7HvY zJ-BmIi&yJ=yb0&%=TCh{%WjRyAiWaWqYayA|CP%2>dQzoiF#l2x}C9E!q)Z0oO0-P z^?n<6**jV)^4H%=?ki<7we{afZ9FIXv><&x4OP@WGm%#`zcpb{vOBW(JzyjCWvzV% zy7)@$5|Gw{hy#6#3qB|B|K8WUwnJg-)$t z+S#vWcf7tl0OL;$d>ujhc()AoMa=H8pA!?QubL3MX?O76aCM%-&{U>TFPs|`ex*oz z(RaD*N`KuE=u3jzN;h8?E92S9t-=rTSH0PQ+)|yu-KD%s`kp1Ind$=C!|}_jdoOP7 zv7Fy9AqrkG9o3Ft<-u&ptKkQ!T<2zkjRjhHPHI$hyz_0GEL2h}61t46+s&mW2+FXn zj#uGybbe;Vaw=#foB`nOsb?0GHj6~Z(N5m<+S%V)@m16_zNw)7U6!{IOc6qAUAOxv zJx!S#NuTFLr2tNh7-gqs&79M|7#>4hPj{zc2FnO4(a=*id zV&y?`Z6i?%ew}He;H(G00U>_IeOV?Ta3`;t%O{_eSVwC{KG#Hw?tI+La$@9aOLHlW zDo8tHY_FolZ)R1-H;W;>1~zO)I-dd=JiU5B$s27WB&cv^>*Z{FRuO&GqcO*9N&P+R zwCGHuu-jp`f9&iGP2QXyBavxM<01g`rUv}Wgb=hBz8rn$Vn1rHI6JwV`?GgF*?Hm_ zz`ZTvB zhY_ti1H~4_P-yG^x8T`(`o^YBZVMLxgPE@UQe)H2x1_-BXI)uPNL;C{a7mBS8!_ns zHv+F7q3u2qGqIbqt<5f9I}4JxW+|A{Qf&?5O8dRCNDtG+yY2w3j?NTD1yCo1~^{y{Mjn&a09D?0= z8?WNY3ezU48Kjuh>R@J`we4G-&1j+!<8{l6TFLZVpYREO(h?uAm#B^ie4KN`h>bSy z=2ih5n(W+g2x4_W4y0GyzE4la!SsSOZ&RO24m!lOUKGC}`%XJTsGv!w=L$Z+^<~@( zQK|?XF4;BF;hYksmoZ>&Q4eG@Tv1^S=pgkQ^5RpPZ`ajo*ri4ox^(n#QW~N)H|uC z00YZ#i2RHM-LR>Q7d7xulpi@%jFM*>mpFV=)LekLQCjNteADPNJ#1bk55I>6yIp$d z?{d^;gwDUzeDrP8R|319Galr|>=udiQZHU1;ikLQLTy<0cxA!93hEYIL@%unmPObo zD|4AHseZgz8u2jAY;4mp*WerDgLm2YPLJmbjDndTuj zT+A{P8A%SRW6;&rkwTU+fT1J-uop~Y=2~j z_{Ydxq~Vp!E8O9lT7&;l+nSeaR>GkB z)?y&l3MlqGshBxvnbLehH`)=8UI5EQWr;zW1(jV>ao|mS$2G|vbE$k$wHTK{qmDz;8LL_zBVU`j7{Qm+uZs5PY%Xf- z&rIDd4e#DHnJz7ipLl%0@%1%OYJtdd`Jj;_2oGzF`}CH&YFs<&BCm35R(0}3ghSnz z=Xop>8Tbo<;}~e5!x=icxMWZ2?I~MirVnxG`n(7q8M6=MtN_PMedzKjnwf5*$G7nv zlhm!}=+5~OS*Ki>`KqbT%lAm`hXjgBtcr5L$GeIvMMU%$w{`XQSq7CWSqJ;%7PXd7C1q7|KLC;WhFOT7QWB*m6@u>hkSd%x=W-iw$f}A+ zO%>tuSwvto<5_wq_VS;raf@`P4zGx-l zRuAi&*(^%0=C(QQHjjNjBoBQ{hxH|=Oxr1vy92ZWEOQG?DQBy-x*Szevxg?qvUm?z zAJU`O17rq0Z`Iv51vNh&C=*kvueu#m!ojp(Ho&TGU)3x1PzJlRJ!2`~Xz3|ibG=gG z)hx2V>r+FXYlVPnbTe%-WlTlKc^L~tS(Y1ly63Hqw?d*ixm=GFwxuW^@+FNu{TTG( z{7cx(6oV&f_42#V?DbZ2@qzEdHwacGS!+6oj94I9$jO&ZjKJ%RcskM#_ZsiK?ww6< zCAQii9^WP=_^?BYy~>4zHKaUiHw=iI7P}(!kx>$%dcWs9Mj|}?-jh<3G-s#F(m7>G z&}M>sIeo!7M}$wf(9E-aRv{Esbp>x}t~trw_Xt4<`PY2UgjVf! z?=^?;>>ouHAKiOK=oQ&KlP#698Lw#FHea43!@d3dX}Xr|Vr^v10$r>yNj%Sa;>Ym< z)8u}{Z?nX81Vd4w(I|D*oO_0Hu;BwjnnS&&FO#`yib*?)h>vXT>DtenR9P=qxD-8} zn3?+lt>9{psJq5SPspGa=Hn9#QF%pbGI`@68J&U>vwk_zl!pvRqwv(j^+d@PMcL_? zCzfWW=JD4zuZvjKWUxR-Xz)?3I;WV!MP;KF1E7|j88RLf)Ea&Y(i*`#??Lhd+6#8c zGw04!Q_sAx*cXA4x%fTI@XeAX#2Yf@0jiVrtUDtU@zP2jQ8$b0xGZ$8+VWqv_Bl1f z+uIeGrpPR?4lx!KCV20=I9BgOW~f0++*nNF$g_C%YgV_#SwT2JBbKBbe43q#%-+UZ zFZdQd)!HjzGAoscXI{0kUMeylSjE0e5nc5P<&nF`EHj!#pao{5;+&_hxq#6Ziev3!|F3 z%MXJh{rKXVMdFny$LunMcYV}890)nK5`xpa^4~NY=1~QE8(C+*L9mJ7VX*fb zwJC~rA`Yhf$1cOC#27PE%SEI(8B$3oJ$*Q~j|uT11&_K%k> zK5c*az)tu!uG6p-=z9F@=IFSYM`0(Gqw*w2hbDQ7>1K@>p0D!Ey}0f(<3TU7-MaFO zPDzaitt{2f=pgGa7ck~9O(^1(o%hTTx+S)TUbRBXQPOojNdosGLJFR&6?&|9^#)f& zo;Ig=gC{p>a*BqW2x2_9>IL5*3$+p#>rM)2Ab?*urGVh!`K)ZE(EKKWr$%A1^dg}b znyTY)MAPmok7!oE0l_uFh(UF#bCGvh=HryxS!T`0Zr^vP+di!rZV-KSL6W@hoEi(N z&f%RrK0a7CHD&K*<+vp8b^elX*QISp5|Pzy>7>?q_J@TM$kEUjA|2VUwf2c1Dt=Ap z8uH8>Tgi5IKr8EBPL6kxi?>R9b*8>f^>Ssz41aCwv`%W539id)*fl`ChM zL9Q6MTMo2)DEj%<_l;f6{wowL5Lcv`Eo0QOs6(sgC&Xq>umv3NqxKt^1GwDoTTpu> zN5fbW)AS3^au~ny`OG;3>mdz?o@EjB9PIyad86M{kzC_vWHPZz7g_GP5ID}1GNdaB zVabwna5%+A<#6WC^YBW;gl7ePi}sN|>sh`15iC12&~J?zE^WIpJ@lu`gcJA3UXRQl za`3Eo%CSs#w(Z`3-Forj&W(#3w(xbZwu^XjoKLnyY0@{$a92lPcKp?sW`hkgOi$s+ zZEC4}s?W~v)2MZRsIPUsaCf&w_tuSfvt^fV^a@!%r5CDOo(^TmAv6ng6HfFcLOFF9 zZF5vjhe@i|-B>B@Oi-lTWdomWe#A62o$7v%? z;?}W}y%eraz5?7kj5a?6Y+Ajx##Xqtcm*586CD^04q zFzET(uoXfA=g5_*QE33MeXoR0U9eqNCTI7aJMO0k$NVzk->Q#3eL5lqKt!Hjdd=?G RV}Gszs48lq%H_>N{vQp8*bV>y literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/goldman-sachs/index.html b/content/ko/case-studies/goldman-sachs/index.html new file mode 100644 index 0000000000..93d3022d12 --- /dev/null +++ b/content/ko/case-studies/goldman-sachs/index.html @@ -0,0 +1,4 @@ +--- +title: Goldman Sachs +content_url: http://blogs.wsj.com/cio/2016/02/24/big-changes-in-goldmans-software-emerge-from-small-containers/ +--- \ No newline at end of file diff --git a/content/ko/case-studies/golfnow/golfnow_featured.png b/content/ko/case-studies/golfnow/golfnow_featured.png new file mode 100644 index 0000000000000000000000000000000000000000..0b99ac3b8f8f81dd990436b54c08f335dbf40996 GIT binary patch literal 22915 zcmeI4cT`hZ+qZ)V3W6vd>0m=j5<&}Asvy0C(n1IjDWQZe2vU?H9q9rhAV?MI9i%EC zD!oaUE>fj_!Ewfn;|$OG*83+}3vy@g-(Ajib~)s%gTIQ>4PwFzga815SWZ?-4fEcL z`H4A=kNK@c-mZ;#Be0RxwFdyskRATu0OAs;0RZSTq=pV!M^Ql#jhRhT4Eb_gb3 zE$LIt>aIGDIWU_KxS3Iy|Tg1~|xs2~W;^y9@MN{F$9?MzGs)ud#8 zq{Dm?VKGOeZ3KZpCnqN^CkPkH&I|~KLZLtqH;|i~6Qkg?cd*bd_9zECI07SQ{e2*5M@)4;j`n{DB<+Z>M*lVx=A8X^px@;; zf&b-W<6vj`9kdA?h_FOhVe;5x7=V8Zz{VVfM%kOA{w3vK-QOYpE-@Nu`Y!+vd;Zl8 z;f(wjx5J)4+`f;gKTfqU28yDhpt2njGyGweQYg5?;c>}HiDEjqOpqpm5>j9gNE$39 z&dn{!CjbTuaC3{xKp@<_P>3W{K%7Sy_^%;;j;J4LeiL3sfR7i0rNF!;~#Sg!vFD`#zwhFQZAKdMGzss{7$7{j1&UQUpJfB+{C2m#@Q zflXnYa5%^W#0vqNfDnIGj^+N3&{7CHq$9#a#tvomeX`iV?CcST16PFQ&yoAr?D|3U z`_LCe{<9_?ZXQVxAB2aOM*wnor~*8a0+J9u2(J`KMjR@|FAV&L>(6>H!G-_mHu>i{ z`VpD;sK{8ZKO_G|_~)VibAZSq?a?SZ7p#uMf)1Atb%gETTmS5^ME;M&{^Ew2>4Gr$ z;ld9h-2!i+p zc|n+uf*{bJrG8iadk}k+DcT8UhY&ZzOfu0wwg#0yR|+Oc$-}LHpXsV1434xuTt3+T zjQe-Te@_56hgq8;OhkcyI{w-4tBWx3@Z?z{|9a`}(8U-deN3zwzCcRfEh2=fABR`|mq{M{A1`&U=p z|8~Xh{?+xTr=~R$Ez12P+b_*nUWeP&A28H0muLu4K3-l3uQ2e}=3hNjkj@B89VsMc z7qdTHNH7!Xhtsb;zxCAp=?Uh=6!|^fcTcRIAD(|q7Ocyu?{n#oR?Nj3<~kAh*M;KW zr`ccY*Z;Bc^N{|J17n9fPJ&J2Nc$Mqk%+Jb$GEU*9BCipIua4K;20M+jU(-2Tt_0p z798Wkrg5ZwjO$25*n(qR*ffr`k8vG|2wQNB3!BD~_A#y_5n&6CabeRq(muv@BqD6V zF)nNxN7~1@jzoklIL3ud<4F4$*O7>@1;@CsX&h-E<2n)%w%`~SHjN|gV_Zif!WJCk z!lrSgeT?f!MA(93T-Y>@w2yHei3nS8j0>B_k@hjJBN1T>j&WhrIMP1GbtEEe!7(mu z8b{j4xQ;}GEjY%7P2))W7}t@Aum#7suxT7=ALBX_5w_qM7dDL}?PFX=BEl9N_902M0Suu%p8VAPe964!7ZSV(Yj*6c1jsIgdd--v~DGDk4uLZqf!E~>wvEO00t zn6t9Tkz9Ok{GuqQ=V_7Fiz3&eoL;R3m1vA9xB9!|jd9G>F80^77RSGVz)&5zO3V&K;%R_rX zzeXW@ijMG8gV!ZoZ||;GH296?XPxD>^b3-j$>%xK`&HRta1i%Ov5Q1boxqK(!JI}V z!C~7XybDe~L7pjgQL*pOwRU7nG@vrgh5Cv5GM56*`%^ylZL&-sSjF?M9IU=LL* zJyn^8Txq14DtXHmikI1yN-en?yHHZbUb;7>81)1(&u%Z&z;Z?nI zjQC8`vovVV+;cDf7~e<)ir5)Ou#F*384%r8C|1kJuf% zADuKg=jktc3M`;z6YQg;JpMo$F1odn-O5-!wbEpIog1mTFd~gtLLV2ycR&i*P+vr} zcWL+JLbu!W0FMAe(1KKwA+3nu=(tCWi$!@0O@f zd7Wa|>h^ssDcirbmgRjLWfL$%WvjA^MhHz zA#Z|4BPi!vXl@0++gPs7FW9K#cTyg)&7OFg%m!yVDOsbWca7cyf?VJC8#2>oR9N`( z;AX&TL~Olor#Z#;hY6Fr()B@)^D*Ad_gtK^lwtOJGwm4 z=;bThXrM3a<#Vr)Dfx@vq*q7!bCDxA_HZ)1!F`HIDMriwxWR?ILBY$JTanELg<}m9 z>-sld;*Fe%WB(asP z&1Lj1fAJ04!`<8&m~MEHE>fBG{8c$3V3Mw$2f)TcE4dMfd`2Exe{v#W&waH3n!4=r zLCfXoeCiUv4n@cMK7FN>3jbS(Nb}3N$C5Dh^hE2St;y2d`k9hf#d{9Zd5j6`bNESO z118zayf_;PS7TL|p||X^hIKS6sGWDcgB@Y3EhvLw-2po88By1XS3~m|)5betrV9cC zK@W1{AI4J#)wU%z+pZ?vwCT^&PHv!T8rBZ#t}ZnCnDBJxvCxKAQOx@+~w!0no0XKhXo5#hDQfBmqRLbi#KKt1oN-YF-sFt z_J4Y{v*?iof}LPH!)VAqVIE<|3gF^={LpCc!V|lu-kVfvvTS~L;xnjWUY8782k~l= zEk-K6n$SL^@aBs0*Gtc4vL*1-iHRIS_A;MQt`KyvodpSK1VcA#e6{&C$AR;0D-qho zV`C_ai5?eU&>pOG*imou<1L~J2bwkup)c<@XD-%MWjOX!#LsErUVrMo1q!4*dsB+# zHjMCBlLS7N9KV= zJozDIH2%%Jt>J;$Ag@*&D+*HMj+V&x(q!CK4eCRWjrzzLqms^{mYyr+S*?Z`C5RIe z;QKX_LHLN2{mY-3tw!@{WV;|O)`gL6OR$U#^`6jADsg$S37Vp2QUNxDoHyY&YPXC~ ziR=~3)03`LVEZIiR5VnzCyVI;sU0ax`b@qWISDeAPL z`(27^un1M>h#(pJre=Gd!1QQL^kV5nNEK|f3*Mi431ak_G^;7~w)c94dIEIMj=E+k zv&YxNcJwjRvy*V5C@aQOq=|d#S0q2LE9!TD3lr7%1mmd%Tn}IuT0Fz*&t}q)7E+mn zkFV*-Plu1+dU5)Z!tn0oqX;;CoM6r}pEPu3tFAtbzIpI&y!GB2QzvQlULCeHvsqp$ zv$Y`4_9ve<(q3$ILKM%>`q9*uo&Y$bZ4Fw-Gw~WUJ6wI+P>`gx=UF@^7`(L zqvBtrpK0(UANG3k^iJHJeWzYj!r%!6VzGkr;tr$BwmaDTvCZ&khGM~S8L z0R^QKW$RlrHz=Iv&7#Nh*f%~Iy4?YD^!Vpqr*;gy|Jj6pK$uy0b2Fz!7CmNW$XT!d z=B&Z)u2b)&(f!q!Fr{cG{n};ENz(drS5YJ9m25YZSxI=TtmrLcqSTDdCdx~ZG=V|Y zR)zhBw{e*=GN?>KW$mWvim&$60~Jcw-}|+>i|d=TjGSQw$a0k3U5lGRJrNyF$z;X1 z(jx-Lu5^q?ZFY3-eNa@_$}4pn;ZmFu(H2HGe3(x>E!Sc#UQ10qG41Bm*^gddtM9*N z)_@<^cqwzXw0@DfYF1RoCc!%# zs#5lpWANZbE@X~?f~@>JG;G4EjuT~V5d(HGp*zd(R#1_J2lcmp%MeBgXpysl$#`&w zr&^Y7ZntiAmiaz>nfV2eSQb#1@4sbEyU@Sl?6@wz<={1X5AGu(iuk%$6Z=io|LLnp z8{~t~6MUMDBkAuyFVT*?+1Mr-uj@C$0nLd}G!Un3p6LoMUhLYLdJ#&Us5f+?cH1}f zHAs5>y;Ixe!U#eVNO2E3Vi+u7AKAD)`6PxC)YOluZ&XI0c+Ysw4) zSFY^Q#ILG94eT_@=%t$J+}{4Uz!{JQAPtqMkMv}(b19V8B>zCZZRS9a{;>Ntmkmzj z$D=y(%Iv0h$(_ASeYnj))fYTVfM-&rUe;?t#f7ZJ*Nh{5{;7)kBFroR|3S(}jZuva z%s$td%#Bcrv>P8^)gjgE%FoQ!g?de%ymDu7PiHG|>I=TZ?$i_2(ZGsb(;FGUH6J5E z>WR~5$hs4E?0w1CucQ)A^sRJw<9R!sAW~zQ^C{VO9z7WDF69bdzidXNb~8vJCi~L~ z+mTECdR=;TwW(=AuRkjkNyNAgelSdm+eB{c44ryV%7dtZ=m=bM=q-KQM`Ko)WayD& zZc_QiUvmHRl-=nMVn2TX6R!=Xrxm~ zMq5JPFq{`@d?!VpMr<#O7RMoexTouy4_atZd11GWs*9@bf?OqqNTN9ib=a$S=;TZ>}Wc5mj=4U)jR2!2KB z$~S(!C3Mmt%i5`TOg_6(`^i`7ka%o5#Ra7547*g0T1|%QeF?Z%^tjv^eQ+3PXjJ+O zntKa}mj+0>$nLbt;#z7$QFClt_V^1H12OourvUmX#S=j*rJ`)2;fc`htfx~3SLHg~ z#;fykA?5tFrbO?FCj7>Q2WN|g?CJ|)>t2uZ->^J;ukX(5E0ypjKwp#dPD^N^t`naB z?Zk6lxqctbZ%>!eO-xbElrp-OoKOqsLxPic`GZ8~#JV z>?dHuYTrIS-F*FW$|6vJpDB${{WPd4^qkeawyB1|jqDHjvi|K+oJQ8PM8gY-dEqZ- z62sud&xuqWiFtGAjfNhR02Xr=LWf~*34JMSo?k`v-${xTWCTd;#PO$XPNv5_P3HXM zmI@&siYJTs@`_o(6Z{H~_G{HxE-qKQ$h7keGb23c%~H6q87Vy6j9B4)$mpR|RoM`CRdM)C zlYYkA7%@eHN>-IG5r{!TB}f)*>0x$wUAPnTMLH$UywnkG_J%Eeg@6fLeST)2lQVBu z)cPf#EE`m=kPXU(ay>wIowHRx|*NgfaqaV+cD= z%*cmAILSiV75cz@Z=e-d6vA;I&u$G8Pp$7bIq%!eZ#M^d+UX%GD=u=VPK*krb@%3H z-YCX_2FaE=P(*~@91UX!w*@=6&94lWym>cC008W7^;iq9ysOx(L)%0`8be7puoBWeHW9BPc^lDG%^@3IT zeL}n&JqtP&Nn0;Isf4LLUmj|@(!j_3EK(n^WfkG!=EKgxE%KTL@x@Gj`6iC1!0ou! z@(B-S^3nwTrKTAwS$%CG0jjF#59yh*4-3Znc~5;w>WsR6+J8R4-gd?vbE#0$S)Zg$ zUn|7EF+Sma<;$b~Z(D=P(-R{v0>AV*1=xNoCAwIVU25%!GoHxj>@BF#qyW$xdmM3I zZ}JqLAYps0$cPTV1 zQai;?e#5tte@0}e&1O&-Gh}4-(RuC0ZFSqhk6X*U^W|-BC){bRJL3!E>x%iP$NGYcJ+#rK0PRraq4TWALsx!ym|2lK%@k#(h{?sk!A z&*Kw4RRHmuctNyffKU0`8@}}yII{ZbS7ZgODAD-nyDC}_*pqa88_5b+*5%KH(#(5$ ze9)*Yw_~i{>}Ju^9})|%vageMzK@T&sE9e3rw~9_;=&yZ01-CPDpY>4D%2+cECiQHiWqS4i zB40P%P(&OZ+}0sB2)_$!7d`XAGE^LO&@~SsN~;zGuYN%tz{j&wp1!v)qZ&=zIBi%y zg^vbC-zO(kjv4{i^R#5&g#>VA5a z<-9pD;f!+Q)r$)-kEY;n=c>Kmk5_tEt;2JSF5lwXLL4DZa_$F`K6=ZzlcD9`rEt6`buh^4Q&c&({%iS^D+4bh{XXRr>RM zAp|t_dwI)@%+D+Z<0mW@377N?dU)hM*R8rj`!eTEXFlpeIhJe--{rJ*CFs zz6eL3tlzWPssbAAdAVwQbYNoc@w#oA(Z`b1s&4=EhW2_qfc&L)=+YK<EN-;>;B}wMZ*tk6UOO=cMX~IA<92qt5fa4ud>F^*h{M zFVC7Df8r`KLFmJCVXY$<0qDe6;Bl#!+dQ!EEr1MXu$e3t9Tf4Zax>_ja|kfyc($vd zTV?$uNyU0Znm@S9*YYDd#Y^S!L4MXRcta{2Ie_H-@iyD$i#++_U-KArF{`{q36m%% zp#;p_d=+**ylZDw@?Z!@(!to|{!1Y1NY4&>Z0loxqGYf5#qq`28yb!8o%+u)M)6dL zPqZaNO6jhryRE@GoCm0sNCCT@qjA&l1g~}7ya|%{*+NDGwq53lT4XU5V`O(u^?aJu zYI}0OJL+Q0BuLWhO5{mOjqEuaq>Mx$|IHVd54Lk&a!Ry4R$t_K9v{^!raqpNi)!@* zxi(}krRuk#qJ{Hl8CrlSTcuU3O0DrtUEh-JD_5#8qHurLDG z0`s}J;^NLz2`b8#Np<@YT*ha+CF|#pEEHOS3gbG6i`wW^+HVs_6!sGD9$Z!iq26{r zHmfSFHg?cf0pgzf{NYW~bMrOJ(xuq&kmX&s*}E{9NbFUCnTNL5nzIg^KQ#0z(0&t$ zP@64tI#DanNFTc~HJY=sn1dXo=6elJ&@&+2co6ipw^%Dm!QDpsYjg7eAuEYx{zJMT z@zWEXY5~3Ye8~-u7%<|M_rs6xUcM5|7D|q|0oMlbi*4YfkM7|I^SI%FoF(xYT+H}O8v@)Y zr}GpaIVdewSY>k}1x_gu(G`>@G1vNhbV<@_5B*H%7)C?6nANr;V#;*MJCx2>PTKnJ zS-t!jIfn9d^SLn7<>H*E_J|#s5eqc+R_ zygLE*gFH^e1?AX#Jf~~m#u7QJt=Mi*?v+p z<<<+6q^6UN3jKP4i67RL$cB|gt`quiy)KrCj*H~HRB2VM;u0h&ujQvRwTFU+uMM>z zN;==xTogmjw+ zBrjTdZ@*dXskcK2;b)h%XgN?*IOYZ-R{H4OHof}z>>J3U%q_qvuQ4u#ZJiM>OK4OM4MzjNi$ID5?;sm&!6?v z@@Ud)M0%ylh~HUzOR>qJL-J6mNfXp@%KdG$!vlkn#jFwO-`s9?UD<37qtUlffXyWnIIC_b7{0Sep6N1RqoL~}W%<<4yOGS8lx=`kw51lBN-O;KIlLdA?^Rw7Skeiz1v>G8DH@UdU5}e)4c=ty) z=CX3`l`l0^0BLzQHLRX9EsdMkA(GCrs8Po}UWwXBDQ*p64+uPosNC5*;TRwMHMJ~$ zutLgI;+8yZ4DWnF$>v^8c6re@W-m~FxVh)cI3-~yD&zEo zDJ4X~Y4MirX65=DYv2=O%_#BcJ|P~5*J4Wv(3MAn8QHX9<3q%`)CxsJF)7i*JDRmT zoIGN{J0GT#I!0H%6iFBEc;xlq1~!{Dm1o~5^YsRw#Y~AMp1dVq7AjrxHzmb2rsW(+mocDzOxLeaW{hDG887MUUz~2DzozpgnD0NqO`pa|;Pr zzMJ}FNI#XI#dcJTEvs7Pgac1!DP3#>$z1F2&KS9m=+Y zAHkA~OqLI`GDQ3Ct%WQv^OyO$b&lVr*YGUEnZ?mopiZB-vHJBRqk$^Vx@C3A89=$j z;^4CWew3@cM(w+WtcKCxI5SFARh(C%otJqZ5rdw)iEcw&7L~h%pQN6urTpcZ3sDym zA*)2!>=oA+_uKZ0=&J`0+}?QLPwRU0ra>NDxsle#adW>6ovpd(CU)1cIiy0NKePSq zZu)v#z0hTnY)j&29qPJYAG{YDQ0PKN-;tjnvuDrqGD&HUW*k}H9iTsDZ2+=cQ_iC{ zMCTgnZJ{m^oYT!Kn*HM6WOd_qw`s`exTE=W46I-b%BnWZ2Hwb=DIXo6?3^Z^Qmhs! zO1z~F%Dz}eFQe+1Em{b=#5R!fNN9ND@LSM0t-M~Odgt@zIbAtxLN)<-kiFmamUFXU zM{nNEQ%;NIRIHA}g^f|1 z4ZIy#Vy1+jyd*ZF`*PDcP?V2-gqvH$bmM@El|X#g@If^w{@-o|V{XU(|J|Scj~m69 rJI?>n`|r1r|D%wfWeNaymjRznm^fOf&xIcTn`Aj@C8=Bqqx=64y0Nx8 literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/golfnow/golfnow_logo.png b/content/ko/case-studies/golfnow/golfnow_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..dbeb127b02a270963c61bb29f21dd9f2d9b576c9 GIT binary patch literal 8858 zcmd6NWmFu&wlxF-1a}DT794_0fZ!Tjf($S?%-|4w2yVd%8iF&p2A2T_hoHfOYjD?( z+;{JN-;a0S@7JrVPgSk6_c?WHudcOvb%chRJQg|`IsyU$mZHLY&1XCN+yc;EJU`Dg zIswn-wVSNIn-<8*%@g1PM3AxsnFDDQ9Rb!rO(4M1+j#&ehJb*?Vymt1rmw0ZVgYjG z1pM{k^l}70qY)6qB)z}@3wxj&jXBWT)=8Z1sHKCB#@14tPVcQMw<=f$Xk)AJ$pxtO zNln}0lf8woC7q-MjhL6nGl3(}4M5}N=-}ik;w4V^4_=Yy{@-aXI+}mDxY>)-{Z~@@ zsv0yhAQvFbTTTHE3m!fp8bM)Bp0~n6y!`AmyxcqjT-?H3Jp3HoJR;n}BHTPQ|Gem) zqq$gGiDQ;0XRVt&{6N%k*3@E-wI>i-(i@ z?~?xIsH*zEyE;1l+uPMm6Zqfx{*T12+TLIwmnP5^73_&8$w>6E-XCneKXKR4BOEojSV<^mk1-L0VCvK*^S)Ju|_ms&oAlDxwntK+-k>7sv4b{Fv zAL>4UwpWKyNxGIge_P%oJDp&U<3Wt2JK+y5oP7xDFzLlf|2qYP|;j=H?j-IFs4OFk+qJ}wDl5~|0jU>&MTThR~l zZGIj*)o+xowhX%Y0>7S@;-yLs<9Lj}Ds84vj!IUq397xhyv z%wMjR{^cW)?rpw5S7fsCG#4?mcK-SFpeKQS@ehA{<&rM6ob1q5LMW_C*<`-@WQxAS__95LLIzrAFqUWsx!_sD zMZ;VVq~j#^d((2PpXfeaE80lmh^6!@#bW38d^o;zyA^=RxjPk3c3bJ^G&|HXdiTO% z>e1qt=#&5ND8-dq7GdrnzZxRVMDhH)Dfii!xdH-r_&ITMnW>l}BrmH4&b~NjicwtUfxCq? zHckSU%^w)*=7P2awC~zj{>dXz5tt`!Bi`GOG8K3Pe`t@Slg9iucTsr=+N0=`jv~Qh z!DsGvmlHztj(KHACibA{-XVJYOGgWh0s|LJWc&5J8x4Vpmkwv7zrKtiQ~HHaSv>HC zCI!>gnGPdls@EoT3neT`7{Yq`ClWofKNb|BRIWy&H0 zY?3<5nVUT}I90N;*~;Wl99pPL$Y6ZeK+X6-~ZyX9zXgh zb#=QbTa}GJltjGn8sS<4N4U(-%YoQ~@9lDYpu_yvG> z>8}MV_u4XTo$;9vHqL9PR8hY&pQXV$IzEoei~e7%gOY~1?KPE93fhD&>L#bd7jm|7`x4- z%2*ueR8&#EugiBob;p(X!pEm&t*70RhM}Uu%8bOb`=VsY$9?~zI7GWhEU;Z$x1IYO zW&4VQ*!AJ+*8cWJYrs*$q)axYypfsC%U47Q@7b;}?~rb#BZV}F)=vIsVTH(WuBZ!SL3rO;@kMANCRs8$=zsB1{NSSd zUH5!SBy16GShm2{6336($%{E6?PM_A?4vuH#O4d$FY&N>)odFi1_;46#!c3Ae#nfj z=c`qkh!!arpGFKjQVDb1FBuirF&-xi_QOrZQbXK#zu(ONG(vYpFss(K9z)-1`QBWt z558Ivvgx@)X=A85qJqY~xbvNTObdaXT?so11DAe*!SUUfUt7&2An5fU!M)9IxG}9J zON$RG^#WfdvTDDdxA~-YTmu~)nXgu1dMs$i-m0MH4-Z58RuacW#z(#%vdnl`rpGzq zgm#)G1vFt)OJ4n%Wl&$I2Uo!nA?=)=h8N3G!$~kxi z^oO4U`WpL5e0qN4lh)LH+`M z#8s8{Jv!eqcrVg+_t^CZ>(MICn{pUc3bxtz{mAt)pJVv|@#7ONuIUoAqSW;!Qo$Ed z@?;^)%`Tth(iQn~J-xUM&!Q5N*RMbASEJZFR&Qp6AjirJj4XjD_m5r;r=@xi;df1Z z+DJdmQ$->&h}iZCd)lvmXf)n}qe2rPq8qmc1E9rPx=@GuA7&$7+E#-AI-Ojnz)&0X4{Zf8ow6ZFS zfU8ZQFmcw0ynfXw#*F%xwsOUp416CkS$0*^>8G>!;kzt*?e=Fl+7~HB?0lp&9~5yv z3PTv?Lq~hh^MOB<+OrsFv9@cRfABJWo7}cbz)9-8mWI_m%V5l@5_`X^YFg>5NlUqd zTDC1+jSm8Tfyiz;w-BY8KY7W zL^n+@6)TB8#Gh#y?0!<1l#D0+RR@82)}Vbj$+ybuL>igafSVx#p}upGT^x^W(TQJU zRo!(N{7U*&?DVTQIh=RBMA8VL@l{6Ws>@d zJCxszsaDIJC8)6$sEZsDOf{3A;vISA$6NWGHoAjI4Ac$`{R9iKrdtQD%$`X4_lcT#J0lgd&@R?KEd~zBtRsU({FLe; zKw++t{_>DikE!LWnV*10$$@YF03TJ;92pDklh#FlqE@pu1J@1hJ9Po{@8`Mp=%@!3 zb0zkOhXmWv1SV$9lz85{lA0KOHBP+>2q4xB?RkOH(kq06a>XSqztbaU*y2sdB zHWD$wknjB~ANlNElz~^{))}ZhT}nZR)SRqSEa4G)H4Sw1qP;X%T>zb!rORW;3Bgr7 zV=w7G8r)iAkUoITrb+oidjE@6tC5{)6PTz8rK@7zcPf&=cf7VM7X|n$oR6iU#>Ma! z0%4#)HPv>XojPaXN2kFkLb)g>{k^SKgFhy4`Lvg1L9OT6;nPa}$SBH+vo7<*2euZ(mM!ZOkdJYALJJrZRLwvpShHB&Z6K}8&Pjv{WJD)g0JM(K;uNa| zh2F5!6jQUag*Rqj6tF4fF!c7xs$szW@$as^zV;}8@`h9>dc}MfiaMJbOjvl!L`3dU zO^XMWZ$Er(OIC>wG2^X+yme`?P@t3Qq<%i`NZ%@XGv@|*-PPa+8~(I&e@Bx2t%r2d z28y7@hA5dKQR(kzG(DRvjVlV*=sUq9myRUkrC1VnQKP~g;Z7>tJ7pw7e3~+kOAFQ2 z6NKGtejX7}$$3uyEWX8o$~%zkuvR%Tk?Vf2$XcYMxb_1DB{a~hy&8nPPLC)pUC^j; zc!2EQYN4-fa$1BW&*)5A<}=aE&%dBqwUv9Np5czZTlJo^sTVbD>}kWjwl-nP;unJ9 z%32-NC%rr-VxcYeU9l_slhm{;4!gJe@Jz0KLMhR~7Gi4L_H`8o+75zfHrDN7u2s1j zW2re$14-<9=oNLG)JTQ$)QjkSHw<*`-i=c2v=uJ0DX+NNSeTH_L_OB@&Fe;fmB{?9 zBIBi_y}C(bHyWBAT7;QxJE~Lt5wfAcmZ8;sf(3LGPabp@jA5+~S7H${{~Iz`p=;tf z3Cjwf(EV@8Sj2=i112VHI??c`p8V6qi$qcxtAzLsq#!VyA(fNqR1gr)6o5fdew3Ik zpR@Inh&sDiFWLzM8>^yiVf+4Zq9q$TzuDD&_XQxIw1hf;$v)7=ZlfgIIEs`vh^hLn zF=v`Znku93n~2fm!<2P$hlvqw$i5K@Mdd;9WHD>4$>!Cqb3!1c-fpt_X_&8W!3b;5 z{l^yr`1P!1;C6LnpW#nP6%y7y%@UOZ9~eC&bB}oW{G6j67Jn||^v(ZEc~t#^(u*XXSWySIplOvA&= zW&tWH**&CFgt3bi0*%1QqsOFN|!*@!LaH(^Cm(bH^>@J#1e}IS0yo=!a=SDe9tLPFh$0bN8kNGHI6WIt%RuH;@`E79Md~ z7QPr+6(L61`#Ii^SOMA8Wsq*oc^71<{1Rp6qQEr^T18x!1!bWpEbux}DEX(ecD55_ zAqV|HeNXK|Ap|KezeGj8yw@}8wLjV8W})Gce`2;B`}5|^2@HuY+JuEI;~PlDG(L(IFne>(1=$I$dd!6L_ry!=@%y z``+`a#vV)x>l-DxhZX6#1tvOdR7DaNrQqyy;TjkRy6aOutF>e44~8ty1}!;vy5Pxm z)4M_s3zufPuzt^vh76LTO`QU!Uq4-m!a88VC3fSV$qvuvPbyy@($?~-3UMZ8ssHlR zjbbvcWHP``Y{bag46RsgX0nIKIX{(_7*QFiet6&fp?=GA!Sk@hdbcxX-^%a$I{oG* zg*LTvaw|LD@o{qj{%yO}%HvWpuF=oAO3_05s~hCROt-M#8siYTzkBbIUTjpk+N^jk ziler7n@}e{`0(_F>b)JEzRMNhjWi#;_M!LTV|- z&!^2gGg)aoyV$i@5le5GmW$?h^r=g}`;j}!<1YiDb4G2sRj^9=wf4;^p@5rIlk1Fof%G zR&_dj?yi%hhw}EVluqt5zqLK~W##5I?Qt#;H&WMdkFg5{?Dk2&ExAHoTuo{6*IRf@ z;m}kAjN>8puGaoIt@GY?5G?>m>wWT>WC|N&iqW>vEOl&R({$z07ka~7?P8;^1PPPG z;nzoEm&dcPxs;!cZsF7AKbNTu-ez#etVy`YLK;P=xZChrqD<{&{v|2om94QyTFGZq z`=SYicbgoijvJ~|)VXUdI6SN{uBKLwn&?aK2ugc^N>X;b`7t3qrk^4@aqGOT5zJC4 z8%(ZdmD562Tb7wx^k9P)Cs7C)MAaitGT~}lS2%0%+ot8po%4eM3u=oBw)M{~4DfQr z$ES|yLfjbqrQ(IJIkJJ79wrJ!SkniVWok2n)wHfT-@%mq0wvN`&EM?2@nLj9u@&*I zA82(c-tebdB+B}FtWHO{`c=FW+BeoHGco;Lkmc^Ib9$(Eu_PnGo`8>1!`>Vc0+~bB zf3m+bTrd5z(NyS)#J+ykuW8?b6*yc$fFite4p)}gT_RoJH^|_5<6ce(C%&2ZQyRsF zdVPsU8|u6MogX`|TdPOD%tVSM!7#Be$$XsAW=do;#Sg3FcJi>Cc>%tl4HhdhDo1}A z`-b0sJ3lI*e*4(L^{*oK} z7IG03V*+t5=Uda#T{Fl{KCP_^o?c2}5tI;YZ6W6987$wBnc9e z&xK;3_-*Em+b5ku?~g$bm+@PLw-Uy7h7mIcX0HrHL$)II%VrZz~Gx$rS)4|9wH2|lwl!mQsfss=1cuH|-|7>91oNVfP`j+0}bV0_QS zxCWV+fSYH#B7+RFmt21MPv&k#xNSAWp!$f2K!nT9wC!lVFiP~1aP{eC9y+6sc>f^R`$J34qut?R3@X^`G zT}!5e^Fu9-5l@GZ7E{BCim@=~P<+a*%-W!(vD&a~5rtND!j$q{D9vY$ZPys0l|IaL zmA)rs!4cr=-Qi%=u6^nmz!;9n^3{;18ijqtra7?a;U|uIxevh8C@whYg*v7S8X|_7 z;-sY8bkYdiNPl&Y_)CHIDd@mS|6y%8F18V?={tLEgT(gPg1w$Z{kmnMjt2(o`yQW4 z7LGv$+_BRFQ3ej(?LZi6gy;->5^xrNRR!U4{<%$;D1l6IcBzZD?-Bs{z!2Vp#G_c< z6Z!yR*V-XK>@4sjr(8{d;B=Qn%npe?9G+}GH2|&M8}&O63s&Zgtrf=Yk~W4KIU`G2 zJ2iOaz>49%Rg7CVdC^#3?Ab{g#loAvK%6N|%vm1<*fUis+?^60=ctFhuY(i2Lx!vN z@Qhm>i$of74pme&r1^Tk{S@*GGHHRjcD{so7QzIhf^$70KWi^vi?43@8-@-R3kqk- zBm~f4S5?Cv`%*(h(7Y!()%Vr8OhWp6@aE9%qlT zdg%4Oe$PkevWzxYB%a(dvdS+=M=O7YEXRiBX*Q^l#9HMjP+b zDfg04(t=cy?AE$ogz*5kcs|BVbtc93tdB|ssKmM3eF5V7lBC9kWGE}Hk;o9E>s~Hq z1{fJ}GQJ$d;rMN`S9_<2xNF(SfMnx?0{2IF?Cag|H_Ma+5)=e_n@fm3viKW2SLP;$hH2NSM%zO! zpquJ+^sm8~BV-)S&tH^YPCsp2-pbjV*Q}Q3aB?V9)1wM6=@;i>dTlfW*jVflDe~Ap z&{?S)Y`7zDcq+=Se`6_6LztR@J%A|fyu4?dmg@bUU!b_29B)3kz}RsWA(5nSq@c-I z^5WN|wJ844D6kAqc6|wPVh_q3<4gz=N3Szf{#pmmsRtZAZtg~qWoo{FbW2}}y<-T%Pjpb??8;Z1Dy3TS-Pe7bF=

Co!VWdh$tGc8PQuMiXrEdE;5v=4p?3 zeU_fOtxdmk=upu%6TT9Mnx+&;ZB+PX<5X&0E?m2^tpENr7kGv~VQ$SJbvE5mxd>K7 z{5JJNnx03|+%vC?qMh$qZznnI&YCe`HEJUTbOz2EY%d%g7*hy|=AO0ppO+(l< z&?AQxhKh`>+BthU6W~>RgX78ZcppS`lY&ves*}$V6Eg#Lu$_RM42~(7qF4^G+0YJa zr8MPN&pKbIoaxms(NkT{&j~Y=tr3Lvtj9?{I)JS|oHHytx$x4V%E7?xV50*vkk+4FYfdndT;L+j{7)Xm()yqi4l%n;ak1$xVx zy+5txfLbhdD-29vkj4DU`P(dPGG?@&E?6zDUJ8U^RLA2Ic>YA$8Mm1NMW**C9%&k- zteHOG3;ud@=^4HtEo&d_8YP@_PQ*MK=SYU)$b*7ckYhkjw5H>h^%~A=;weWbhG5QYklHdGHQho~8CT}kcTb0dJg(nq8rC{fUm>(QY{&4QXc2bbC? zKNOoEi=)5M(rE1m=56IoC1XVKp2tDiUh!U4<3bxujT-JuD4TvU*lPK(c#y9=lz^_O zLQq-m+0%IS@v3XD0b6ic^5Ak3bE8JCykAY5xX>n5eW2Y<2|Xgt(W~ixc;!lP#&gcw zTw4uEQi&`y?%*CVr8DSD5lPY<`pi=z)Jkn6Ja3HtG562gBdohSqE9DjNJfe7o`&W# zFHKH(vI&-2<|(&ZGkmcO#~-w2oT9H+jaQ>3@<bA|N50A{82n^kg>w{aZv)R_%R-v>D|8 E0r1&*@&Et; literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/golfnow/index.html b/content/ko/case-studies/golfnow/index.html new file mode 100644 index 0000000000..f4bf4d4f27 --- /dev/null +++ b/content/ko/case-studies/golfnow/index.html @@ -0,0 +1,125 @@ +--- +title: GolfNow Case Study + +case_study_styles: true +cid: caseStudies +css: /css/style_golfnow.css +--- + +

+

CASE STUDY:
+
Saving Time and Money with Cloud Native Infrastructure
+

+
+ +
+ Company GolfNow     Location Orlando, Florida     Industry Golf Industry Technology and Services Provider +
+ +
+ +
+
+
+ +

Challenge

+ A member of the NBC Sports Group, GolfNow is the golf industry’s technology and services leader, managing 10 different products, as well as the largest e-commerce tee time marketplace in the world. As its business began expanding rapidly and globally, GolfNow’s monolithic application became problematic. "We kept growing our infrastructure vertically rather than horizontally, and the cost of doing business became problematic," says Sheriff Mohamed, GolfNow’s Director, Architecture. "We wanted the ability to more easily expand globally." +
+
+ +
+

Solution

+ Turning to microservices and containerization, GolfNow began moving its applications and databases from third-party services to its own clusters running on Docker and Kubernetes.

+ +

Impact

+ The results were immediate. While maintaining the same capacity—and beyond, during peak periods—GolfNow saw its infrastructure costs for the first application virtually cut in half. +
+
+
+ + +
+
+ "With our growth we obviously needed to expand our infrastructure, and we kept growing vertically rather than horizontally. We were basically wasting money and doubling the cost of our infrastructure."

- SHERIFF MOHAMED, DIRECTOR, ARCHITECTURE AT GOLFNOW +
+
+ +
+
+

It’s not every day that you can say you’ve slashed an operating expense by half.

+ + But Sheriff Mohamed and Josh Chandler did just that when they helped lead their company, GolfNow, on a journey from a monolithic to a containerized, cloud native infrastructure managed by Kubernetes. +

+ A top-performing business within the NBC Sports Group, GolfNow is a technology and services company with the largest tee time marketplace in the world. GolfNow serves 5 million active golfers across 10 different products. In recent years, the business had grown so fast that the infrastructure supporting their giant monolithic application (written in C#.NET and backed by SQL Server database management system) could not keep up. "With our growth we obviously needed to expand our infrastructure, and we kept growing vertically rather than horizontally," says Sheriff, GolfNow’s Director, Architecture. "Our costs were growing exponentially. And on top of that, we had to build a Disaster Recovery (DR) environment, which then meant we’d have to copy exactly what we had in our original data center to another data center that was just the standby. We were basically wasting money and doubling the cost of our infrastructure." +

+ In moving just the first of GolfNow’s important applications—a booking engine for golf courses and B2B marketing platform—from third-party services to their own Kubernetes environment, "our bill went down drastically," says Sheriff. +

+ The path to those stellar results began in late 2014. In order to support GolfNow’s global growth, the team decided that the company needed to have multiple data centers and the ability to quickly and easily re-route traffic as needed. "From there we knew that we needed to go in a direction of breaking things apart, microservices, and containerization," says Sheriff. "At the time we were trying to get away from C#.NET and SQL Server since it didn’t run very well on Linux, where everything container was running smoothly." +

+ To that end, the team shifted to working with Node.js, the open-source, cross-platform JavaScript runtime environment for developing tools and applications, and MongoDB, the open-source database program. At the time, Docker, the platform for deploying applications in containers, was still new. But once the team began experimenting with it, Sheriff says, "we realized that was the way we wanted to go, especially since that’s the way the industry is heading." +
+
+ +
+
+ "The team migrated the rest of the application into their Kubernetes cluster. And the impact was immediate: On top of cutting monthly costs by a large percentage, says Sheriff, 'Running at the same capacity and during our peak time, we were able to horizontally grow. Since we were using our VMs more efficiently with containers, we didn’t have to pay extra money at all.'" +
+
+ +
+
+ GolfNow’s dev team ran an "internal, low-key" proof of concept and were won over. "We really liked how easy it was to be able to pass containers around to each other and have them up and running in no time, exactly the way it was running on my machine," says Sheriff. "Because that is always the biggest gripe that Ops has with developers, right? ‘It worked on my machine!’ But then we started getting to the point of, ‘How do we make sure that these things stay up and running?’"

+ That led the team on a quest to find the right orchestration system for the company’s needs. Sheriff says the first few options they tried were either too heavy or "didn’t feel quite right." In late summer 2015, they discovered the just-released Kubernetes, which Sheriff immediately liked for its ease of use. "We did another proof of concept," he says, "and Kubernetes won because of the fact that the community backing was there, built on top of what Google had already done." +

+ But before they could go with Kubernetes, NBC, GolfNow’s parent company, also asked them to comparison shop with another company. Sheriff and his team liked the competing company’s platform user interface, but didn’t like that its platform would not allow containers to run natively on Docker. With no clear decision in sight, Sheriff’s VP at GolfNow, Steve McElwee, set up a three-month trial during which a GolfNow team (consisting of Sheriff and Josh, who’s now Lead Architect, Open Platforms) would build out a Kubernetes environment, and a large NBC team would build out one with the other company’s platform. +

+ "We spun up the cluster and we tried to get everything to run the way we wanted it to run," Sheriff says. "The biggest thing that we took away from it is that not only did we want our applications to run within Kubernetes and Docker, we also wanted our databases to run there. We literally wanted our entire infrastructure to run within Kubernetes." +

+ At the time there was nothing in the community to help them get Kafka and MongoDB clusters running within a Kubernetes and Docker environment, so Sheriff and Josh figured it out on their own, taking a full month to get it right. "Everything started rolling from there," Sheriff says. "We were able to get all our applications connected, and we finished our side of the proof of concept a month in advance. My VP was like, ‘Alright, it’s over. Kubernetes wins.’" +

+ The next step, beginning in January 2016, was getting everything working in production. The team focused first on one application that was already written in Node.js and MongoDB. A booking engine for golf courses and B2B marketing platform, the application was already going in the microservice direction but wasn’t quite finished yet. At the time, it was running in Heroku Compose and other third-party services—resulting in a large monthly bill. + +
+
+ +
+
+ "'The time I spent actually moving the applications was under 30 seconds! We can move data centers in just incredible amounts of time. If you haven’t come from the Kubernetes world you wouldn’t believe me.' Sheriff puts it in these terms: 'Before Kubernetes I wasn’t sleeping at night, literally. I was woken up all the time, because things were down. After Kubernetes, I’ve been sleeping at night.'" +
+
+ +
+
+ "The goal was to take all of that out and put it within this new platform we’ve created with Kubernetes on Google Compute Engine (GCE)," says Sheriff. "So we ended up building piece by piece, in parallel, what was out in Heroku and Compose, in our Kubernetes cluster. Then, literally, just switched configs in the background. So in Heroku we had the app running hitting a Compose database. We’d take the config, change it and make it hit the database that was running in our cluster." +

+ Using this procedure, they were able to migrate piecemeal, without any downtime. The first migration was done during off hours, but to test the limits, the team migrated the second database in the middle of the day, when lots of users were running the application. "We did it," Sheriff says, "and again it was successful. Nobody noticed." +

+ After three weeks of monitoring to make sure everything was running stable, the team migrated the rest of the application into their Kubernetes cluster. And the impact was immediate: On top of cutting monthly costs by a large percentage, says Sheriff, "Running at the same capacity and during our peak time, we were able to horizontally grow. Since we were using our VMs more efficiently with containers, we didn’t have to pay extra money at all." +

+ Not only were they saving money, but they were also saving time. "I had a meeting this morning about migrating some applications from one cluster to another," says Josh. "I spent about 2 hours explaining the process. The time I spent actually moving the applications was under 30 seconds! We can move data centers in just incredible amounts of time. If you haven’t come from the Kubernetes world you wouldn’t believe me." Sheriff puts it in these terms: "Before Kubernetes I wasn’t sleeping at night, literally. I was woken up all the time, because things were down. After Kubernetes, I’ve been sleeping at night." +

+ A small percentage of the applications on GolfNow have been migrated over to the Kubernetes environment. "Our Core Team is rewriting a lot of the .NET applications into .NET Core [which is compatible with Linux and Docker] so that we can run them within containers," says Sheriff. +

+ Looking ahead, Sheriff and his team want to spend 2017 continuing to build a whole platform around Kubernetes with Drone, an open-source continuous delivery platform, to make it more developer-centric. "Now they’re able to manage configuration, they’re able to manage their deployments and things like that, making all these subteams that are now creating all these microservices, be self sufficient," he says. "So it can pull us away from applications and allow us to just make sure the cluster is running and healthy, and then actually migrate that over to our Ops team." + +
+
+ +
+
+ "Having gone from complete newbies to production-ready in three months, the GolfNow team is eager to encourage other companies to follow their lead. 'This is The Six Million Dollar Man of the cloud right now,' adds Josh. 'Just try it out, watch it happen. I feel like the proof is in the pudding when you look at these kinds of application stacks. They’re faster, they’re more resilient.'" +
+
+ +
+
+ And long-term, Sheriff has an even bigger goal for getting more people into the Kubernetes fold. "We’re actually trying to make this platform generic enough so that any of our sister companies can use it if they wish," he says. "Most definitely I think it can be used as a model. I think the way we migrated into it, the way we built it out, are all ways that I think other companies can learn from, and should not be afraid of." +

+ The GolfNow team is also giving back to the Kubernetes community by open-sourcing a bot framework that Josh built. "We noticed that the dashboard user interface is actually moving a lot faster than when we started," says Sheriff. "However we realized what we needed was something that’s more of a bot that really helps us administer Kubernetes as a whole through Slack." Josh explains: "With the Kubernetes-Slack integration, you can essentially hook into a cluster and the issue commands and edit configurations. We’ve tried to simplify the security configuration as much as possible. We hope this will be our major thank you to Kubernetes, for everything you’ve given us." +

+ Having gone from complete newbies to production-ready in three months, the GolfNow team is eager to encourage other companies to follow their lead. The lessons they’ve learned: "You’ve got to have buy-in from your boss," says Sheriff. "Another big deal is having two to three people dedicated to this type of endeavor. You can’t have people who are half in, half out." And if you don’t have buy-in from the get go, proving it out will get you there. +

+ "This is The Six Million Dollar Man of the cloud right now," adds Josh. "Just try it out, watch it happen. I feel like the proof is in the pudding when you look at these kinds of application stacks. They’re faster, they’re more resilient." + +
+
diff --git a/content/ko/case-studies/haufegroup/haufegroup_featured.png b/content/ko/case-studies/haufegroup/haufegroup_featured.png new file mode 100644 index 0000000000000000000000000000000000000000..08b09ec9db8b7792d07a9a4d385cbd546d2446ed GIT binary patch literal 5841 zcmb_gXH-*b)<#hTR8VPGP>4Z#2`!=b-jOOT2qZv2l9+@dLGX%7vjs(Z4^0F_I-w}i zI|5QdKw78)3r*^Wcf2#-TI0;GIqNy+ea~L|+3#*Y&e@R`X8LEDxtZzc=*}7%=vW@D z?MHV!Bg4_tjJ@0LXt{{fy^ga&dEf$EF>pFf7|IO}F!Xixgj>R0VYhF!!&T_$=!Fs1 z*KyZPO_bbGzLKs#btD6Q(MN1LIu-Rmw5z)h90zcNdm@mkz%^0>5P*QG0`0)2Ql@Ba zxEI182m`kYGP8CM@^M#$0oBz2DuGHz27KW-S3sceO(a$+P!;%>U8SS_PcaAx_)7)n zqYC_oQ`b!`0NN-F8~~PtNVrSONde>)C8Z%!a$sq3fQ*#1JV*)*l7dJ`Nh?VyD9K0z zeqF$$(=ad(B}*N>UuPY?sRF%lIJ6Q76c7*~86Yc(!gzwD6%`dhQZgVJ8Hpne3G8ho z&NWa1iRJ&>f({((jzOSt2ow_V)1s>z$`7XsJo5A(A^4(AP5)^aiT#zRqm+RHUC|(E zNhy%8@6WjY(#GN};s0{ux7t|i+h{n*5{^arVcd`E;lckm`6ze){h*(UN8Tvupxpg@ z;YggJjwF2%EaPBalCZB0m%S z%U?JK;SY!DVNkw+zcQqR_>C9RieNc8MR~Bijwbl82y|s-b@cS4^<==ZU^yKf;4d!h zH_`v%LjEHcbmR=^XLd7zZ=-AW^bu_I5 z$5s-VaMtjbttgtYX#Q1L{{8#n_cdT=UD*it2N*?>!V0{OeF^8!qsLC_Ts}!#%_@BC zOApD@9(XNr{GskOM+8jY|M(SgL%_2K=L|$01PO81{Y7jfeEpd--z9)JsU{|R(D z$NwGgKxg1SLYph2&l@nK--uDKj6SxCiWqgJy2CEOD?Q#MVIX@(%j6~xk55Hw!ir$R})gK@ih z$3)NCQ_P~M_`3ZbW#xPK?#VbnLsi6lTD^Ho2WBFxEiJV$#mO*?J&8-x2_Ycgi_-i0 z=%TD!aIZ~bHz^L}$m!%@`>B$c7XdSqH@}w*yL=s1ZDO_|seGpb=^Uw*u@EH7FoRR- zy}6HEjukLI@FWg-^Ben0h`cabB8-Q<>Y_#o7{{faVPY~o)Qup%dDB-OvOjN)s*Hi= zQw{x}-7AdO#bj*|(%O{FCiiCeVybTPa$Up)1^Msb8o2m6J1l*FEEPNnNamVocDp7L zyD$G1Fbod~iE73tt$_su1SU**Ak3vgU>^MCXjt33Kg~D|lgUI5^6D)+R2~TUwE@X$*2He9D{1hMHM(@b0B)#SWIfj> zc5?lqR?<+-Bo$qyfVog?|Hv}`443gj>z_8!{|v7YIHYX z-_EE@;(#I&$5X#<{;O@VCR%7pm5rBWZFDs!i{9*6GoOG-*k z{osP4f2<3A(W40oUR@0h*?HvJ$Y|g7QF?QS-u|ks3n4IR2gr{&lk%;8PgxpSd|1+H z6YTWC?ZQ_6+#o+tC0Wh6a#h3fDe>6lIxY}08TFz=Jygv;ksWG1bw-CT#|*!@@Orcq z@9!qO5)R8q88ld3M+_iAVxZi~I_cct8x9T*_)n3$jx|?gtw?UoM)t0n@7pAuYx-K@ZQ9qqVY$Z$HY^ti9hY?z;268(8i@>C9FAE^Dst z+u7YYTx9=!@nE6ISvL4)5qhU=Q8S$7QhK*zy$U-U8#Y`4>g-$>I>9%;vAelkTzA)N6rH;pIl|sJh&P_*IhYJk3(kzD^jx#b%84KOL!#HJ%@jftk&poID2bh% zEHIy}f8yBZA14}BfRxg zR31W(J8K&!_q1SdD9+R6$M($A-O{Y_-ES+SK_@-WB{zOj`SOl_k>^w4X|<3gYfk*0 zVE9bCQjlB?u}X)SIl*}k#{m^)RK@R<DU z_2NQ5|HYQOo1+Co-V_@g*ALrymzCFk-As}?sy3~_z!7*oWcz8al zo@s__5|}-+mSf6pPZas+Q#l4>RoPCD8F=lZvRUHZY9_gjc5O+*QWIV9GaJf@Po7>> zera3maA$q^dFmhA^Q>x&iTfR5-D;6m0jHyf(yj9y^js6-58u7hm{x<}i!>xTOs+T! z3kWnfPG1tesCo!dRZYQQFq`YE6nqmutLh+X|KVBkkIF11&;>I^poyST@GPV1*Ae$D zoySw42Ny&@;qsVo_mVO*+nO0B7+7q7Oc*wV1i!FjCqjEVX^;*vg~_4^#)|}TraY0t zn0DA9hH&BpgN1Hv0=532<;E3{r>(}c(YI_Hc7AYh_ET=%G}Kt1cT>z)&rT{RM&1)? zvf8$qU@+WK%_%Hz=X8U9>5aj4#+dKX(a~`wS(cFQjk(Ua(`i9V`Dgm;vtLE2tN?vCn?1I0$RbxU@^VJ0@%1<#e@%6VGGUvpp4re)(eQ?%(UjRdtE@xwa?iz&*%XDw zmFWsSi2jf*b&wp->B8g2z4BoW6Iq{h$Et@&ByKa!@Ig?cWs11|Mpi56o&5$BkTU(A zIqZzwA5GmWuXPS7vst2#^*qk7e5$&?h5CTDhsC+zH-@e$diGcb8su3t%+hw&SLS$C zU=gx4j<1Dl4ME;dTKt<%pvkz|FRp&no!+?}I@NZRTP4+&-%{f&e!QtkDo(G!FLJ$& zoT~Jsm5b9OTT`3CdA1>_!gfB=-~zcYNm*RH3sE|nUq>*)Kwomp^I&rWQfs0472oAJ zn=cRugve-4iLw0Iu`gN8(e7N5@o6IxBG-s@Ih-G|ICI3_(vzF9CtJ|&Z@wJ-8FF8> z%F(H|yr=?_zWz}_ewa_b93qYjJ$?RzmWY*)T$4y#HM~Lj>IK(`IOb@t=`1#{gX!km z+>Zrcg9S?CC3_D)haC$I+`VkZ=Qd_E1w)?F7v?m0)-Q}Pk?j9xh0>8-&%DGkN7D2ku|u+ z4en;~tfbC8R+%FCajEIMbaCrspx+9BVoRDNOuVYq)X|)}n`pU{Bsxx$43`6>m~oLm zRU{Y~Rx<*VI&XjQH79t;W%~jx^Zgu_Mhm!c%4qpDv$=|;$|;M#U4qno%tDHdodc-!;rTM5U{$6q$`18=H`KVRWYVB)z^EgTeD zn_C}}p^wgl}Wn7%G zKVYTZP6a6Q4F!|X9f|~#Fmp>bA{|Dm?AIX{hyW@RvEzPYW1s$D+o@ZiG2Xyok7fQiCVGJz1F{;x&=X0wn zYO*-ohq-|n?4Ch zLgzS5KFiR3*s{qc+7r3Fw{dTHr{=>^Ng$pfsYQd{3yb-^v)?lESR#q-5p|}$JWA)L zb%LlF4^_&}TKOv`v%GqppOb!rpLI-QZjM84bro;fGOh~o9O=s-PNT(|_*cas^m)>A z+acsF5e#%NNB2Ckk2bGp&~UIHmYM`E-J=CeuoTQkK8}jY{1$~74ORI8YiPQEVrgi% z;(26bBzrom7h|2+nbv0Wo7;0lAM~@H)Zqb=gOmS#3xqSKpYr7jG}?*|IWThYvR@?D4`ucA-IEbw#q1XyqVuxQcMvJ>|-?^@nr_@}M`WD$XS1T=lIx(Vh72 z1m)axI}9tiH7)qUbLf<*x$5bY2+l{w2kwp`tO zz3U}+!?Z+Pr?!Cr>dlFM=$jmbc3yTLa`Nq5Cm%FnPQa4LM)ELja4OzldzR{QW8;ToZ}j<>kBz9?j&gsrt5 z@Rs4R9yQ5=;g&6WrR_Y$B~E-4b%td#S?#vWl!5K1J_;#dVDHEG?}xOK)~151g*lEg zB+6=l=4obTf7(QKC!jX|%OAE$u|b%)#f?|?+ zFR9l{ii@w9sW-3o@9hPPN{VMr$GFv8NB^n%Ll<)^$il`Z_SKLV@tjy~#)|+^b|FQa zl2+hn%Jg);&u|et??ugg`cq?BCOKuc-j9tSua>mKSz=$rzFc~8NzusjJ7#gO8cWF{ z`Gdb5hY)XxM;-l)aZ1);{eoTkE4J_J>R24tPc18xJC&>UA^1`H=VerVzDn=@uwilA zNH(RCiA``s&VlyD@x0l{496^$YD=A;GbAtHJQE`wKm?vQ>!A7H&NaCs_~a}t(KL>J zUUZAM3vA4D(QWHB-WouO^NylA#yhl286AE6d}HNR@q*E#)FzY#nz6(2(zx9(Q`|+4 zczGxuP8ln2o-R)@d@iJ{;LAfT?w+2WzRV+!lnW{Ix3IGV(DxRcf9*qV_c-T>pXY zef%lb%RvG{%H$8=l(~{tzRGd@T3tEeEQGC(x>qh3lXe_<_1lRm;#hWuFxwjMld1=_ z=t?(`oG)(#9$gkO=T2eV6K_)^)a|GC2R!ChUY!`r?mXZeNr1dipF8%ppCy*Lq~+fB zVfud2IMObm7yC!E`9k&SD)4UDw4&Lal)0Ml$jhT;$3J^U8<-n++li=y8U~JOMj}qA zdjhMVDQfNv0)kibxaASbypKXUz)W=Nk!5^VbpJj*`6c<^&4P|*ME|>4(9w+Ox3i!S e*S%sodb+1#Q>rIbPtN}wG8^ie=@e<*xc7ezU4{Ap literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/haufegroup/haufegroup_logo.png b/content/ko/case-studies/haufegroup/haufegroup_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..5d8245b0f6d18fcf18d871f7c657716b37f2ef96 GIT binary patch literal 5794 zcmcgwcT`hZx5ok~MMOaq5FtuaAhZxd6-f{Tf;2@y5J+eN0wIJBib~4>(pv=S0s_*z z8L0w>-s{j62puB*#TjS5_s93Xf8KiUu6^%0cdz~1XSaLqSv%w*{5~@SCj$))4YP(i zRF66iQE&fqbkwJhLD!8yLD|FE!U~1N@>(Ko?3@$C|e_a6TY8ZsIoE}u|*ILv!MSv|9>nsNZy1ToJxl4(mF*ZO62m}HYmjp^mic%#+ zT|J$!79OHbt~dYI07bf5VeFi-c4#NwpBgPJ(Qa5p0M*jJTyS*O*8WGalk2ZQQ9}mw zuy6)Sh=~Io9e?`ur?e|p5BV=M{;jmDfu}PPsE2e#yJ4)T`LMqEH<%i`|GdyoL8>)! zP_&hsBhm@00aXN0IbsMq1T{LUFj=SyR1K^OmXLsfA(HoGRm7$5se&OgU}-6-zp)xl zu2>5vE9Bo;yMJL-|07mT6@#?EqA><&w8P)oduWTsqFrs#&b+FsKeGkm71Fk_vUB>$ z6#g0DU-lv~b~q$L4TE;%{WC&xcK^l-2?$798UhA^p(>z1J%C9`LDkeG)FeStAZaKR z@C%FhH{Sok%KVR5Ak`S)&-DJcbpOqw*3i%CKgCOZ`KSDmPSl#mP>WkX;0Q=VbNQkM zRK>t!bS3G$hk;>Ss}5Q7cJ3`i?yFZH&H_-wuv^opv+Q6tZe^41L{?VkQCjGAnD<^; zdGYn%Pr2D9mBW7jC}e*pY~>4__Z9#7vn3Z#YasLcKYgtR;$i3bLhL@)o+uO3zvfd` zYdiK*T~|GI7Mx{$&eD9t&Ly0t;T8KyG)=U>=y?^*Uqo|};}_A)oc=|3O=KJ}QSib{xyoqC-oEdTyOhm^qT6c9F?Pr8 z?02H1q@>#34Mz1X^3|K^6xHpHR`mDx>ymVood5WJk22{Co_yC6*)>F`!?zXLy2zjP zNK?0`r@!{`npK3BN2Xdv&$uV)DvrEAJqx=rv5JFe2zyVR5pKikMFv|lvKpTUyJl*7YT6@gu4kZ+_aAG-;EIx&k*+ zyDXv1EhK=xaO{!H$2UXPetuCf|Kt-0KR^Gcux0SM>NiO+z1A5{$!E{;%>(w*6p1Ov zk$cpH@YmmiJ>n}Ao8BHQ@}KZoDPMUN4kyS;WEAElahtK&%`;#&DsyfIqFmn_nQHr+ zh{WBD7SyhBJ)n$zX>-i;_{tC$JF)ov4d2JU^^z(%2Xzk32RI#{?VcB}IT*a@=(!4{ zY)I{45%`-yeJb?3H!jZfaxu|?07J~(_T^^!F(KSCT-O2uzWJDmy{4nDoYylKPZj(C zmU$sKx%vhWuQstsm{PKR}+qJe+s*~eJz^}<*bmoXdHjhSIuLM2ieu3VT=o-N>b_V`b zS1Z0?iP7Z`Sc)RWbXxf)S46y7ozcO>)UjII#`)g*`jv5RTy3d$1e@lwXBY3Y+9VI^ zeGVwfEnsh0`dE^emxp8Fl0>|!P7lDXj+EB>8yjDu}F2uB%zUWxuHd_$c2jaPiQOsT~BT4&?- zHrY!W0sA|s zDntA2-E@e%U%M2J$`rvs^IET+oP+dQ9BI9@y5W0YHvRQ+VycYe@#Zrvt*_T-Jlx!N zK2`fCj8Vplh=YOH^V%%4JGlkr-SZ`qobD&SAYlD=uWopS9r&r~@wq;x>EI?tHt(1E zRvGE(ZDYIH$<%*$Be@V**mO2)Y~JZo-PLy_OIEWf=e~THU(eU+KZFW&^|c&+i(3?) z6O%Sny{nq;r^Rj%!}9GCMUB3sGX2?<7? zKKBmQiMNaDCEP9ncGs^GN9#t|czB8@Iu1xO9H7iyMQQAC_X~&d7v$?4YftO#!{YQV zUCM(*F|AJpJ}DS9?E&*~ZiwpX96O1KwE?svm4ZlO-hX_ik0o3%W9JL!QS`Dr4q}v? zOxnucD+ikqGd6xC6kFL1v>V>{gM1G;b+YxDUZvFBGpj9r<4R)#04!vfx1eh-u%Ik1 z{-!EIn+N!&QabyMSWHrqy@f7I?Agyl**Q7-aCqmcS-n@7$6Q*_exV7`kIP(sX#tPd z&o~zya^bp?mX5;iuuyT3z3{+gVZ|T|OmSL5diobqgDtI~cdTNx-`)?Q)ZzCB9#hjF z_02w4%j^i3dxzetYuVNjh@QZ>(4M7x2#Zae^f?Oq+;HBbUddkbyuk`E|4zQJZ@viLjVWyt)?lr%9*`lY2ygY$WLoA((xvO`7^57%_&*M|%T z%)&9C{SyQ4c#$W)VK+zNPAG&v8?lqH&)G}KS2eEpt(tJ^aWD0^ClCnS$`*%>3nVV^ zYcbPxi>3Uc6N;!hqi57BK6*~3^ymwZ{5l$EGj4~eSu3Zm){(fC1%AGuskz=#|hOh9OQ*k=4idS)tz3qK1?n><-mg{;pg$JgEEZxzEf(Ar6 zuE;$5;YLA$z-o|jfk~#zr-~JQPag{l8BS?LH|=qL>&|X50Mm?HUO3Ol=sH5aV2?t? z5K^*Q8Uf#&`MVaQcMB&2Enh$p2*gaoYesh=&$WY|bXYOtLYL-B96_kpRljsvS1Oug zB>1Rdk~pRMd=)KJUeIDAp18HBgNg(z9L%TfB(P1#!;wcaPUi zfnlZ%&w*F2^fcWU?5>01H>EkXhj$d{@$0>HEK1)_4l1Xnr3JIg4fGdWFi?aA?X@sK zkbpk)Y=QB?$mHDWguXEzTDenGV7n6<`x5K*bQwYvKPT+D?_&0(X@=#giQe=sfPwS; z#Ek8C^GB->v>rSl-&M)bL>;kSJL|ots-vvPUszpS;o@(*^ksi}+bi)mbgOV}5-erF zh=1-_?G`_ONGMm9P3GDOb}zK)9j}#1>uq<>@yvmc#bhP4=IlVR)y^=tXn0xaCL62U z>skA8ukomWz(#V6z`dNl=88w3*o4983vH{oiI0aOy=g7m)U$OtTvO3&7eP-Bb08Z} zeoX26z73WU=zQB$Gr)1(r&l05ug3UQjoe9|CEho>@}&8S#Fd~%wMc>Jx{Sc_B<78l zl!lvt6$O##57A%o%$25WXZ-5Yv=}mvanW&cYWsIz3d%1hOe5go<vXFfD$|J7Ogpm9l;wq?PG zOzDAw*&vDms!;s?xNSlhySQhgl!%1I%*lgLd=?mJobh>VqT9W-xm7s7y+gx>DJrED zc0cJ`=>RH#j)^Bqai)`1&~weB)eV{s+Eg2!MVUuJgY~xY_3w%gwvxtOWQ7;%if1x<#T92j|&|AzQ{vv_(6hAkVjOr}jgI zr?~^&efp_w^QyYkY7J{yL#gN0k)!nXB%Qzk;sj>F@lZ=>>rg@(UGBj0UqGH|+6 z6mzXsd6K0ZzCF})G3JE|uHFf~)&GXmm$-9a0O3-dwPdJVwws1Ln({jeNJ$2j9+KTB zFO+TuKw&UlPgwPea*9t0+VbA%ofWsSkftUmLpObY4qctBwJR^Q=sRiqsxm2a;he59*e=O^3V~JtI@;Jq1D5iQ0Yj4c8Wr_9RZ zMhUcx9JdYq*yZcZa*A?uSz#ic_-V>$AZoKFH|d>td34lqSB}eHiLnUwGcP zFelw6R&SxJuYU)=2`dj@!5x4c0|NuI z(c@Jvtw)rpRHc2645^j~f7{dn`!Euz>&7Rq9sB(Ww4XNy;pLTXZv8BQG9Ba{ktzrA zS@@c$TNkJssTG+3gI=kqK^?Jrx{PV_VQ(Y4xFGa zONa$whYFucqbKVQ{k>NtHkNdQdhaDRPTNu}IoQ}%#>=g8R6Mk&(;RSIJKuU#6<%I6 z3=JOS!LJ-l2bK69P@-Qj56o_C&a{93yfM4n(9p2zkao`bma=kcC?o!TRc3-18)5N_ z%ZqVBi-DnGmK}&wnEm49^(97I2Z}}7~CT3=GQA34nJl9n0&d|xGYh}L}8qqW? zk$7}N!Q*4*hcW~AAM31g(eav#ohjO(tD^&77|m0iN)p84VvH_DM`7H2%zG++yn;a= zp1xo6;tahU1je;sqd;r!zok77StY`~*gN?dpi; zas9}u0kG`~*OUcEw33r0cM@e~y7NPI54s+$Lf)X_7!=Yu6WQ3+DOri>dsniT5z_qB zmfIfv1#a;Y@uYCd1MQtjkTuhUl2272Wxk%Xv%U^TO0wMW8qREseoHXBfXXG!%*>Fv zWE{l(%A9pG#qOd>S0N?1r82wln3x!){)RgbWs=E1l0c~O<)NbZ#rgTy=f=KGLdf{arY`Um!zM*px#rDcr?bbxUzAen$tgYV zVVv@-lcW5aC)~p$J&02RORmqBuU*J}7uXPUb`94eS;^{wZ sh;ES?(ecQh|0z0}WWqG#+A{O}&#tis3=S>4_blMQ0Oo0{wEzGB literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/haufegroup/index.html b/content/ko/case-studies/haufegroup/index.html new file mode 100644 index 0000000000..f4256ff569 --- /dev/null +++ b/content/ko/case-studies/haufegroup/index.html @@ -0,0 +1,112 @@ +--- +title: Haufe Group Case Study + +case_study_styles: true +cid: caseStudies +css: /css/style_haufegroup.css +--- + + +
+

CASE STUDY:
Paving the Way for Cloud Native for Midsize Companies

+ +
+ +
+ Company  Haufe Group     Location  Freiburg, Germany     Industry  Media and Software +
+ +
+ +
+ +
+
+

Challenge

+ Founded in 1930 as a traditional publisher, Haufe Group has grown into a media and software company with 95 percent of its sales from digital products. Over the years, the company has gone from having "hardware in the basement" to outsourcing its infrastructure operations and IT. More recently, the development of new products, from Internet portals for tax experts to personnel training software, has created demands for increased speed, reliability and scalability. "We need to be able to move faster," says Solution Architect Martin Danielsson. "Adapting workloads is something that we really want to be able to do." +
+
+

Solution

+ Haufe Group began its cloud-native journey when Microsoft Azure became available in Europe; the company needed cloud deployments for its desktop apps with bandwidth-heavy download services. "After that, it has been different projects trying out different things," says Danielsson. Two years ago, Holger Reinhardt joined Haufe Group as CTO and rapidly re-oriented the traditional host provider-based approach toward a cloud and API-first strategy. +
+
+ A core part of this strategy was a strong mandate to embrace infrastructure-as-code across the entire software deployment lifecycle via Docker. The company is now getting ready to go live with two services in production using Kubernetes orchestration on Microsoft Azure and Amazon Web Services. The team is also working on breaking up one of their core Java Enterprise desktop products into microservices to allow for better evolvability and dynamic scaling in the cloud. +
+
+

Impact

+ With the ability to adapt workloads, Danielsson says, teams "will be able to scale down to around half the capacity at night, saving 30 percent of the hardware cost." Plus, shorter release times have had a major impact. "Before, we had to announce at least a week in advance when we wanted to do a release because there was a huge checklist of things that you had to do," he says. "By going cloud native, we have the infrastructure in place to be able to automate all of these things. Now we can get a new release done in half an hour instead of days." + +
+
+ +
+ +
+
+ "Over the next couple of years, people won’t even think that much about it when they want to run containers. Kubernetes is going to be the go-to solution."

- Martin Danielsson, Solution Architect, Haufe Group +
+
+ +
+ +
+

More than 80 years ago, Haufe Group was founded as a traditional publishing company, printing books and commentary on paper.

By the 1990s, though, the company’s leaders recognized that the future was digital, and to their credit, were able to transform Haufe Group into a media and software business that now gets 95 percent of its sales from digital products. "Among the German companies doing this, we were one of the early adopters," says Martin Danielsson, Solution Architect for Haufe Group.

+ And now they’re leading the way for midsize companies embracing cloud-native technology like Kubernetes. "The really big companies like Ticketmaster and Google get it right, and the startups get it right because they’re faster," says Danielsson. "We’re in this big lump of companies in the middle with a lot of legacy, a lot of structure, a lot of culture that does not easily fit the cloud technologies. We’re just 1,500 people, but we have hundreds of customer-facing applications. So we’re doing things that will be relevant for many companies of our size or even smaller."

+ Many of those legacy challenges stemmed from simply following the technology trends of the times. "We used to do full DevOps," he says. In the 1990s and 2000s, "that meant that you had your hardware in the basement. And then 10 years ago, the hype of the moment was to outsource application operations, outsource everything, and strip down your IT department to take away the distraction of all these hardware things. That’s not our area of expertise. We didn’t want to be an infrastructure provider. And now comes the backlash of that."

+ Haufe Group began feeling the pain as they were developing more new products, from Internet portals for tax experts to personnel training software, that have created demands for increased speed, reliability and scalability. "Right now, we have this break in workflows, where we go from writing concepts to developing, handing it over to production and then handing that over to your host provider," he says. "And then when things go bad we have no clue what went wrong. We definitely want to take back control, and we want to move a lot faster. Adapting workloads is something that we really want to be able to do."

+ Those needs led them to explore cloud-native technology. Their first foray into the cloud was doing deployments in Microsoft Azure, once it became available in Europe, for desktop products that had built-in download services. Hosting expenses for such bandwidth-heavy services were too high, so the company turned to the cloud. "After that, it has been different projects trying out different things," says Danielsson. +
+
+ +
+
+ "We have been doing containers for the last two years, and we really got the hang of how they work," says Danielsson. "But it was always for development and test, never in production, because we didn’t fully understand how that would work. And to me, Kubernetes was definitely the technology that solved that." +
+
+ +
+
+ + Two years ago, Holger Reinhardt joined Haufe Group as CTO and rapidly re-oriented the traditional host provider-based approach toward a cloud and API-first strategy. A core part of this strategy was a strong mandate to embrace infrastructure-as-code across the entire software deployment lifecycle via Docker. + Some experiments went further than others; German regulations about sensitive data proved to be a road block in moving some workloads to Azure and Amazon Web Services. "Due to our history, Germany is really strict with things like personally identifiable data," Danielsson says.

+ These experiments took on new life with the arrival of the Azure Sovereign Cloud for Germany (an Azure clone run by the German T-Systems provider). With the availability of Azure.de—which conforms to Germany’s privacy regulations—teams started to seriously consider deploying production loads in Docker into the cloud. "We have been doing containers for the last two years, and we really got the hang of how they work," says Danielsson. "But it was always for development and test, never in production, because we didn’t fully understand how that would work. And to me, Kubernetes was definitely the technology that solved that."

+ In parallel, Danielsson had built an API management system with the aim of supporting CI/CD scenarios, aspects of which were missing in off-the-shelf API management products. With a foundation based on Mashape’s Kong gateway, it is open-sourced as wicked.haufe.io. He put wicked.haufe.io to use with his product team.

Otherwise, Danielsson says his philosophy was "don’t try to reinvent the wheel all the time. Go for what’s there and 99 percent of the time it will be enough. And if you think you really need something custom or additional, think perhaps once or twice again. One of the things that I find so amazing with this cloud-native framework is that everything ties in."

+ Currently, Haufe Group is working on two projects using Kubernetes in production. One is a new mobile application for researching legislation and tax laws. "We needed a way to take out functionality from a legacy core and put an application on top of that with an API gateway—a lot of moving parts that screams containers," says Danielsson. So the team moved the build pipeline away from "deploying to some old, huge machine that you could deploy anything to" and onto a Kubernetes cluster where there would be automatic CI/CD "with feature branches and all these things that were a bit tedious in the past." +
+
+ +
+
+ "Before, we had to announce at least a week in advance when we wanted to do a release because there was a huge checklist of things that you had to do," says Danielsson. "By going cloud native, we have the infrastructure in place to be able to automate all of these things. Now we can get a new release done in half an hour instead of days." +
+
+ +
+
+ It was a proof of concept effort, and the proof was in the pudding. "Everyone was really impressed at what we accomplished in a week," says Danielsson. "We did these kinds of integrations just to make sure that we got a handle on how Kubernetes works. If you can create optimism and buzz around something, it’s half won. And if the developers and project managers know this is working, you’re more or less done." Adds Reinhardt: "You need to create some very visible, quick wins in order to overcome the status quo."

+ The impact on the speed of deployment was clear: "Before, we had to announce at least a week in advance when we wanted to do a release because there was a huge checklist of things that you had to do," says Danielsson. "By going cloud native, we have the infrastructure in place to be able to automate all of these things. Now we can get a new release done in half an hour instead of days."

+ The potential impact on cost was another bonus. "Hosting applications is quite expensive, so moving to the cloud is something that we really want to be able to do," says Danielsson. With the ability to adapt workloads, teams "will be able to scale down to around half the capacity at night, saving 30 percent of the hardware cost."

+ Just as importantly, Danielsson says, there’s added flexibility: "When we try to move or rework applications that are really crucial, it’s often tricky to validate whether the path we want to take is going to work out well. In order to validate that, we would need to reproduce the environment and really do testing, and that’s prohibitively expensive and simply not doable with traditional host providers. Cloud native gives us the ability to do risky changes and validate them in a cost-effective way."

+ As word of the two successful test projects spread throughout the company, interest in Kubernetes has grown. "We want to be able to support our developers in running Kubernetes clusters but we’re not there yet, so we allow them to do it as long as they’re aware that they are on their own," says Danielsson. "So that’s why we are also looking at things like [the managed Kubernetes platform] CoreOS Tectonic, Azure Container Service, ECS, etc. These kinds of services will be a lot more relevant to midsize companies that want to leverage cloud native but don’t have the IT departments or the structure around that."

+ In the next year and a half, Danielsson says the company will be working on moving one of their legacy desktop products, a web app for researching legislation and tax laws originally built in Java Enterprise, onto cloud-native technology. "We’re doing a microservice split out right now so that we can independently deploy the different parts," he says. The main website, which provides free content for customers, is also moving to cloud native. + +
+
+ +
+
+ "the execution of a strategy requires alignment of culture, structure and technology. Only if those three dimensions are aligned can you successfully execute a transformation into microservices and cloud-native architectures. And it is only then that the Cloud will pay the dividends in much faster speeds in product innovation and much lower operational costs." + +
+
+ +
+
+ But with these goals, Danielsson believes there are bigger cultural challenges that need to be constantly addressed. The move to new technology, not to mention a shift toward DevOps, means a lot of change for employees. "The roles were rather fixed in the past," he says. "You had developers, you had project leads, you had testers. And now you get into these really, really important things like test automation. Testers aren’t actually doing click testing anymore, and they have to write automated testing. And if you really want to go full-blown CI/CD, all these little pieces have to work together so that you get the confidence to do a check in, and know this check in is going to land in production, because if I messed up, some test is going to break. This is a really powerful thing because whatever you do, whenever you merge something into the trunk or to the master, this is going live. And that’s where you either get the people or they run away screaming." + Danielsson understands that it may take some people much longer to get used to the new ways.

+ "Culture is nothing that you can force on people," he says. "You have to live it for yourself. You have to evangelize. You have to show the advantages time and time again: This is how you can do it, this is what you get from it." To that end, his team has scheduled daylong workshops for the staff, bringing in outside experts to talk about everything from API to Devops to cloud.

+ For every person who runs away screaming, many others get drawn in. "Get that foot in the door and make them really interested in this stuff," says Danielsson. "Usually it catches on. We have people you never would have expected chanting, ‘Docker Docker Docker’ now. It’s cool to see them realize that there is a world outside of their Python libraries. It’s awesome to see them really work with Kubernetes."

+ Ultimately, Reinhardt says, "the execution of a strategy requires alignment of culture, structure and technology. Only if those three dimensions are aligned can you successfully execute a transformation into microservices and cloud-native architectures. And it is only then that the Cloud will pay the dividends in much faster speeds in product innovation and much lower operational costs." + +
+
diff --git a/content/ko/case-studies/homeoffice/homeoffice_logo.png b/content/ko/case-studies/homeoffice/homeoffice_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..35d97226111593770ce84fe1a73597fee9273457 GIT binary patch literal 27411 zcmeHvc|4U{`}b{ZBWy#4%rejOJV#Q9BqYSPGtb+YG9(!`Nsr^kuJ0~fPgj$a_%tyDL8Mye)h~nJ z1K@7}3>RE+O_P_vFCy3TCLRz(LUr&L2TDqzgCH3pyDP>hV;yZdgo~4qrL~I{Qpm^2 z71V|x1!W&sON0Xw#cGALv2#{r|5n$;&T40^$ZjO5BckJa7HMmD-p?Ir;HP^9;pc#m zwPsgVB3AH`106UaQI@PePL9qVaz2Xezx68z$_KZF*;#+9f^tw~S3MYz)mTT5^{k6K zl2uYjN)RC`F2*V?DUrQ?&Pn04%`@ukee*SnbC)Ym*a`yNGJAk6FkEN@ysE~;8 zKQ*#O{Mp9U)7|m6saYe0k&Z|wq%+C`v=jZucCNNAC>IY~m;W;4pO63Rz+g>ubp9Os zALr}j^p8V(pftPy1b;yKk1ai}__`v6FC#r%JlzpU4KMI2Xa4E!QFb@~TAY9D>EOwK zojcOU?!PoUc=CI*-$dc}6;c52DtFc$X^C=izvANJsPwy(>HP~r*0X01q)Jjql9flt z5@F|jfGn?q@ZX#LwdQ~Ih*Y;kA(akfQcy$|{9F;0k`t4b6Bid0k(Lt?In>GTHUHL! zj*GS3P2Yd$Lqb$eQu=Tohid-451>QVmMF{rRWFC@9_q&$A$QZo-N_QAWangQgA{gk zwowrN>*k@Fe@d3zSrNjbV({lDeSvn(-TIx#RhLD||wVag* zQbr7kkQJ1al@<|{KuC%U%19%v1TAHeB1lnjDWtXN&EG)O(K+1wKe|?TL3kc$%pYA_ z|6g`}sQG_%eaYPpSUXF{Lv;>j|9?BgKXmgSL!Gzt05a-(NOpk)|7On&kZ%8e_g@bj z?S8XMu9of|$OElYWdE1i{b|Vlv{MIde$!q#OT>XiQ$idl1=3nU_}}jSHLX8e{?W|t z|8)U>@BdKq|2~GdEz-{uzefJOzVX4jI(q(AUs_BA ze1t_0u75N={P^!pP5$SmhadmF>0d1ko$XLcVtA!yX5cDJClQN29+#`Nx(ff3*}9lRn(?(33w}{=Qg;Hum4v z^7p&IZ2%sN@SiTrzc2Hj>hS-$@YgH-fA08jcgKSqM&roxF$AE!*wJ?hbtVzbr_8!%g1mX3DMyS$8a4+y&#$8a49(cucma2-bD$nr5$AE!*wJ?hbtVzbr_8!%l{)>#D9GW5a|rQ`R5J3 z;upM><^{eU$ci|BSqFmrI3XzTE(C4wgWvNI`E~Y*D3`mY1J!h)S>uCFKR4ObNwcpMWq{p5QT(U?b|6?!}BnU+yw1 zwx#p9LouY2>^}PN2kiIVMO@j4LgAA4IPfVyEn=MuJN@^49%-D6$Y90U*jbq#ERu7W z6;H43kQY~vN3KK|mAue%IpxzzqF(y@CFskkQXg;;hBtiv?Lv@gRR8(lqOulYB&7F- zf5$JT4I!0W>PRm$7xG}-D=U*HlShs*nk8UYKQ=ooEUczRCZQc!W=x03TN@l494Sh-f3D4UPqoG5 z#eF6qWOB#SBWSTTsCz$zJx0t9-i{gFIn;N@f{r<0Xx+J z28O5D23Ea7X73r?XiG4^b!}i#(gN3uT7zOxx8Jj3u!!e^qvKM6VWE`Em;wTWe(5nc zFB{5)%{OJ>s65p0-Vzb$W8>aC?sWfO>;3o1n24cZ)?GgKfJ=JwBMfzo?`h0^CJBU>?&NF8 z&c%JA;?`Bz9Pu#7)qeN(?aJn+5j7_#=gjt8x{IA1uaJ;e-Dr&RUQOE+5gD0z#Z4Ut z2L}{t4Peq&WI{zjF}dpVzP&wF*T&E9Cnk!)(~L#J_JKfqXz7oOVcVTG+k*_k3wB2M zU|Yp(44-7_yKfV^_z z*}KNuJ62X!pQomPD0(Uwnwd2?^p|Xo2Q<~zp4P#9`SN9J%co*+2_<(R2@4Agq6p0H z6wCg3U0sZ6=t=r=``!W!=A4E;CbwnZ*ONezqLA*syp}e z?EME_MOPQU&$Ra!n@WSHZ_;@pA~Hw(7Ka_wzhV6cZY_q<#lfl+N3in_^ecPDP7dn9V#UlZIj}yDxGxbB5;~988tTUZUGQ4$ z)6tLH{xN5C^=jkVr;9T)Gjw$?XW~sIbnqbUyEXpVKD|-vW)*0k{A+5l9BSwN{deO9 zO`YpkF>TDZmO>ev2O4kRa&s%=(7L4j3k8I{4AJK9;gOYyGK-1R1{${SA4dWwl@_<+QZCroAfRqO=vVk+1WqOCcEtQ zH|_OiX~sx8^g$|3cYZn7&gX`Nge(tNOG`=$n$=3Mu}uPDtMmR=HR!lGcB_;pqO!7b zf2&Oytd7!+=0I^9>`5lMp0`hG7^7|*wz1mvvpp9wmPd;+)=Z_eVKB}kHEuvlhpSz> zIy;%ftlxfkdRAOqe1CUNnMKj>MRv9|Sc09+Zx&UKgW$E4eyw$*(P&@~icBhQw#N%H zNIAkzo;(SIeeh&@#HKfYYdKvgFRAWEA)PqR^GB}O{!$ATX68FTI;FEH(P3dL>+4|d zrEOFChLXU9@bU3UNmUgT@CgVAaB^zELjnT>LqkKml=uC=Pro~RpRARERC?yqHTQ%C zJM-R=nYg-MdqZjh4{F^sg?W_R#fulyeLjb~G;R|@%y+gXIWNXfeB0mc+Ak_By!CTF z59dbDGksA}(Z&8!8}z8}Txw%u^lV z=9+)^r%#z#SqFk3jscdzZ|#e|xw(+mr!OC~y1Tp0g!&(_ou$^01^0n{lXe+PNljf| zTSxsYNM_>?6jWWCKs`Fcie4SVXe9Mk8zW-d##NpW9GON)`YFW3(a@7=q{eJSmV zp(GdAY}&1_;W06^8gAc7@x51_)B_#n>8(Q`|wshOFLE?19#otv9;m7znBE3WBz zd2IyjZw3gP)%LWt&AFeMlGdiKFTd%K;CdBX9DI>E@fRsMM+i?u`JG=s?YeU<7c9-q z$MOyGz)Ph0E#w2|0W5rcLPDuY#q7%DYb7NmAXQ*5eYspU3^0EA{P}Yr!Avr)tUzk` z_zLp#=cDDnCQ8^{2K%as$zTxPiF476+l4AG7ch~fZoRvOq+Z*o_mi*hpVD{<3^P#Y z{f%Ll_VOiH8C*QP0N~Fx?X8#ItT3T&1?mcfUsGLO1T*4XJ?i{zylHrCLC35S>4eq@hqZIM;>{m#LKh@ltH z1@@C%VMWE!(b3vEJClbwrp>-mH(5$hK|z6=+XiS4SYh)jM-zAV@=<4Rxzy_MJ3IBh zKQ0>>aBC+%h>xH3s>{=U0)W3~uv)qeAMl)(wBVvHDENtCa1p^cH*U{rMAF>eS*{Ld z9a!G@^y!n%KKcL=fwA6B83V*DXs4@0DC&-4iv=i?dG(y*6M~CSYDKBPUmTDP4*G zYLTl9qi)_0&}MJtHF?wi-p+#u53;kfO}3~(Jm@FNkYQU&kwe?aj>Z!7+IZU zDJEHW>+SC|K>BN}3{+KLfs+WAZc1W80(ZMRkTF_+v$yA#F0-9HBTqt`MnX&scBZA? zLZfeE4Nbe>+kqaOIB`OrF&eyAn&K@?Tbq%gp|1jSM<-i|goMONIuTajlL32mduJn6 z!Y;O{X+MtFAe4;B4|wQ6U!xeLh2-ULwMLu*?wZVM55IZu?vFUfY>X@Rw!c3RAg^y@ zqhQg287gi3X%DtuU-hYi{WKNTeN50=>8Ao{j5d5>(@uXn;+O_AQJ@Ry$h22qH$>cK+CHeKzNdplW{Vh zKAnsdZVU?#2g>cz^7=IuCFK}E3iwgi1mbgZxm4kfQVe7~c+G7r0Rrvr`uZP%Kd#Ny z{Y2d6=lWt^!2)pT!H0=V^UfWmTziX)I46YR!ez!Hk#{d|bbd zyiCe%Sno3h6^5Gv0)WE`yy6m5s<$L>BDq3MonPhv4_Z*L7}yiAaR6^%b}B}>5m?B+ zzCK`-PSMhKrO5tx6HdkTNGl$x3NI)uBxh0DO=#NXl$YNbc4-HvJLl?4NfPg@#e5SOU*-n-l=;fhrp33&l9Q4G57e|4H4hFA;}tet2kwx0 zgCB5fNoa-8k&!#V&&$(E?(69ZSqyD$ZB^Xg{Q<1{zV~d$lgvzZW#y)#B7vB;ZZKoz zU-Oq%S69Qs!@*9ko96-gFsJr%PAXC8^KZQ?Ut7}O*EiRi<6^~w7zQW6qHW_2_W zzUr595>CG$pTQu{vY6J}xg1(a(k^M4JisxF$dpOdHS15m^K-tuy!`X$&k5#!yh=)q z85zyv;{ii2?0S290WQG3Rr38l-B)a?$lTTOMBJ#zc=f|s@&j0b3ko*gSmdC!n3={y z{L(0&LH@4_imwvg^6ZAd(nhs&V$&2&yI;h`#Q|?!JC*~SB!F=$HE#e8VmcWyvHQ#G zdwY9e375tic7SuE8O=C2G?bK(APszDO_tj|_LbmNGKu~bos4Vdsg#mHv9I^~p@(ya zLvEauw6w7C^9*p3E-Y<)-_fCf`U3x2EC!Bhz(B})%!b^%cY9;d1{gkIaGuoxZ{Nei zgH!8$SJx$7-8}%R_tz2L_VPi;#%J_70ndw=p zsAX&G1NLZepdNN>xL*=}vNUR-mlWctC@L!IH@~G5KLd0GoR*>)q@91vc6!WqE< zeQVqSZr#UHizaX?!V4UA4w!vn(+K7Z{;3SM$EQ!9)}S^d9T$9pwltrCU`MVUP_6D4 zpq=xpV1n)KZJ_A4CW1}?BXw|05^mgL=jNUR$4Wt7-tC?7{jm=pL>vd64_4Sr$;l*4 zA%vgyNmZ#gusb~Upko?$AMdCcI)Lp-Y8XTq#=s#T zR0p91Fw_;X8ixwE_R`Fi2>$#07rV@0VMg)Hcg~8lswQ>L0u2Iw%tEcXo?c*hxY3zm zU~lHtI$iZ&WH0*J+CBsOnV{e@kbbaf7#MU%Gs%NsfVDAhKl=8P=ha$0UpY6zpDjSh zyhlBa+}uPVd;)@BGr|E+WIb}*XT*{_=e!@uH!OZHUbs_c{cSx-c!}nd7s+IZD=uST zisF`OK_J!CJH{a?I4cA-l}k|TI&G}30y!A3_D;vHfzO*a>K*aTPZ!dIo3}o0&|rP< zGx^!JRoE~I;D%7$P=T6WOi3~-r?`Ia{^Z5CZTMVZp!bA2(-p!L+*2y7a!!TT#Aq>@ zVLqGTtnL6=OQvhl;*X~Y)FHUa89vAcO{K>3^dxE`1eeoEL}rH;qMJ(*#lGaxI@c|( z!yN*@fDUtX5pbc2jxH%zkaL^5;B7SRrC4*HD zgG$ZMHVbqoL*t!M;pIeoK$swpf`ts+rz*T6WYG1D6k@qAwHOY2N*Q+!)od!!G91)NqiE9!v|wG#+IKhmq&2%iBeqAz{710D{*Ic&1iPt^anyriV~^H38AXnab$9?WDu?_3D<6im*+*9z-H_p%d_}|a%o$d3Bhj|hdh~8guo%SYGYv0>%9=ekhca8@0iO-RH|5h8Usj8h#>RL;)^;liM1*qLRH zPTBq-dP2O(IK50!Qb=M#$n4zrz8XxT)hV4jdwWs=7q4qOx?azKbB5f%@3~0BFn$Ir zjUl8VQ`2mxlYwqr#)W2_$Sx%tkwS%& z*z)ZBOR#~nWjJU*z;~p2F(`w_UR4=&y)=clsHn80RKEL4oauG5Yi34+&uXZ3`j|SL za{E+xRq)@bYdpLV0%LzDBho^Y6X*V3WeVnx@%%valp5yZ@KtvdM=McJO;)pE{9L;V z&lJ>*!lEXGMsqwmv-pK>O5Z&%@Md<3if)GG6z!A55%F?OV;@Ij>+9KqebuB7lXN{V zsAoFx<>0z9@eteyxy-6=B%wy2mXC8TxQb&Hcl8tAL$N5g_Zn1&sC#ay25^-dXewg% z#}Yb{;Wu&dB~~PS6+ADvnVU?K-<#RCu`f8ypex-YyGXzudcl(Y6c0sFPn*$;`4qPa zv_l6($%YmWg2llYWe|K248LuW4v&lSW7tbL>jzNj)-~4d*;25V@+><{_I1Bag%cc;%D~Sa&L9sC3J7wEM6l7 zU7tG5Lu|nDPMrt>W(X_8rs7k#iF!zwNgGB>sGcg8N;U7+2}WSO-|XSyQ}e}!BsNiBHr-nQ?XsaS5w6&Zhiv8 z%~5%&62_{h&(4lh;AS?(=f}{jI;U+}-eUMT|74;qWpe>{E@yW1`3rQ3AL?9Ocl{y6Dr@oEf$*8J% znqQmqZp=C}9uhrIseRR6QlG@NMa8CBQfLuP6jGCD+GkCAC2v;D;E_prSS%|n(gU%OJ8$C&)qs(k^4*=A&0n*)b6i22YV!AbK0mA$iT{}a31|fCL-ck z&l^|mry#UAxwpKeX5p;u$J4+3abIWM{q?%2T9@gGJHloEM!?1A1h7D2MwG@ToI*SE z%}dDKX2zQ|pKb7il+i;EvK03y5-#GCfw(f~UwsTloM`Oo*eY3S>n<>@3|D=1`FVH} z0i1|4#UqN}d35Hc7!ErmY+w?33^?saIT=AE|5Lsv`@ql6haBJ>q|9viOiLUCL+TLD@X-S?7tWst=5K zWN4~SG~1KP;VZX-25`bs%gzT4^<1LV zW~{-k(G*u|ct>!(I&J;c>+EzfRp^^Mjke8*6a4WO22Y6bIoK(<;Gwk^+;I;!YL-rE zH@{p`Q3#-X|59Z-uD#$UB#~~f_+rQPV>M?R`J7R&;k_s9f|>8Q z@Xz;5_n4+8owY15bASu@`GX_lXja{31j#oFntpL1Ii(@#5#rhCwTloUgy zQ_3FSald@`U%P}=ir)QnS0O`k0m+HXi)>HEzKBn_`gl6Mpy-*CXC-0S-qlxDkJJd* z=NYYM?erF434)np`FraPoH$Q$aVBD?-EpSey;CU+E+=u@1qds|8VCi15G1YeCr zY9c2K1rOAdX3wK=EvaFDfPBBhp}?g|uSw4U#+ixfz1lmLImnB=qpwspnpw9VG2h3X zd#NeyBT*TRGQTmY;alof9ej2eyS+fxSXVl(hrUff-tv0IrNsv+7)e8J*05Z!y{pZL z%`5ReX*@i@)VRr>03-S|PRbkvHeGjeeOKup~<$$JA2`sIm56`0s@XJzTr-0;i2HS8Xze zn>bQ;{kzr`q8IK76YV?M+?`K$SspgDe5SM#i7#|BB#gmGK7?fi!MIduiEtiBXhFaV zQp0etFR`5PXSyBqqnyPn!aD}6q{4ezI-vOP9`g4mz#osii(~)CD6N| z`O+>9E~MS>`2cY_^YV%8>?rN8;v)KJ3T*9*(}<9Hbwr4PTHqOCf*bhi7#Rnat+{4d zh>cqOj0n-9ebRM*sdr&%W-r32umqUlHp(D_hZjR4Rp@9@WX(b1X5B=ta01+qybr;T zo%i}w@vgd;i8*GKMJ!Tx-=ggG6#m9VvJuasq+zjmSqi#N*#}dOp5)Am1B{Niko4Q@ zhB}$*o9n8nGs<=YWqZ@y$lc-Pa&?E2z zZ%MpzB&2|9Fw{}@=tbP8TsH}HSfT6966&c`n4uou;W?N(JaOw$?jqc6+pcJ~K#kSq z;U7-39o)_|O!HbNP;#GG-l{4APF zY7F6cY8VvkO!9H{N&b~w87jZF+w;~j zk2F|lb`Xag`efI$zFGIq=R=}$fnCy|BO#=rPpyFiRScpRqimBfbW|D?XT-_{3@+$! zQ8~XWpGc`qVmH;&Ufu&b$$kN2&4P37s041986fP}CzjmlZT`fS3(4dEz}+XBgFwWT81ma# zlD)OzI5M!4u<+31OB|J-zGxf?K6?Z!tcFYX_U&IFYQOPq+~WFmN2!CXuqL2g0M-Ve zvBb2!rmHX(h{(Lb3TP?hN?SKT)C6QjGI_ueUH)s$6d+4TBzJ-o9n0;tkKho_NOqqU zMiYs|rpSUMMPR|ROpd+Tuc$JM+o`w|{8k_k0wUs2AjtHAgm2*)c!HqmMvuWI92I-} z1rR(2!2zKHX5no$me8gdlpKf}fIibexCn%zAQdnz4lO=j-jAM&AUy_#PEouiRbm)i z=y6LS4J05yK$-yp;T1D}O)KpRk#MWpcb8>TUf_Ng!ovo)O zi^_G`-7^*ZC9hX>l0n{Kf*eHhbl0uQjNu75I5>1kp%$jZJD2q*e3@HiH#axOn*#Rn z$Q1$^Rs~I~96=-wa$T8t1p+qd3claNxzpept02Y~k9^|cTxJ|EWER20C}H~n#8*Jp z8>D4H!H< zUlI3H5No@1K|B9dwC$Cr1in(1tM$DF)g*|b?$=#SC7x{+bLL#I;MbGJFKi#S&)^0g zq&L9Cgc919y~a`9Iw1NaB!s92B#GT!|3~jQA~FjKY8=FK*6z`ohebzUrOvVGO7k9Z z(*$WZ05wleEJwq}kV8;VP`SY7^6+NWu=67hA_^~OOS#AUBWhkVGd{IEaMG|8h5)5a z7((07aD>JLj<24FqvxL_`pp=QUq3LgO)+%BOCG5+d-@+l0jd~U1 ztSl-9hDc?!K@G8V3HK{m@4C@*L{X`5S5Lhq7bfMPs*6Srt3m}yq$ll?`ofn_X3_AH z&cCuMyK?n64g}x>kljBcE9(mu3`C1zs;cnDov(Eun6IG^GGhoUtBs|>3S@gc$jpH3 zj=YRaBK-2@%OIubr*jY?16g4ae_)p?Ua)Jpx_IYP)9iNt9Jc{u*|^7fgQWv>10u7GnI8f--b zs?YNCRp*AR#hgZKB{?|omFcBfLEyHnjayMM_40yO-R;}Ae|-N=#;jlrw6n~Z5Ev@= z>9-#~=oN|XhSb;BgP{3HVzJA3BT!f(7_8Z9%BsVt@!bqRUs^hqNj#te5DAO{Qic3z zx!3Zwr4MH`qaB@{F{XVbX32?(VSB3X?qU!=KE8lS`J1WLn(IO)@tSD5Aaam48h!tm zKdS2+$Q6NH-K)DNUbnQYgn{ghP5N!F_VVueIOH}6r~j2?+>rnn|Tc0!_;YmqF)y zBOYCot6<##DA0K=t!)rOMw*nAlvE8br=I>C!TPk&<%E%iMU&-gLUh?A$l^q!%0qy? zQ+DVRFf;_ojYj)kK2S3^i zNE0D`{<_IfhUY$Dzbbf@Ae2DD>-+2Hy775nkbmuLPIY6@^morW0q#eeJ+H7Z`N@-U z97GAH#iId*67_A2$*vM8H5iDGj~D{PMgudm#H-m#QoAvgoRVo7*ao^@k)nTm}dPLRA$9Qdd{c zFpK%wezSt@>Fo40-SE=00y)ox9%+%C9LqkIY17V$^m*SDz zAY1$0N5S?Rm?Na(=~-E_cy1Rq8mx;DM2o7zUAx44T5RJdJI+=QJJrZLR}Vi&^X8D< zKdVn|Xh_5c()wLpg8r)z z4g>^{X-XAr$f>wfZ&SA<@FdP_$QT1-IOa1I1;{HN7lUQGJr#lVELMS<-;&_IUC{!m ziIp4b>i7liL*l;-wZ= z6Iwo|Zp@u~o_Sg7ZAVRmqGUQ0n4ZoeH3!%vAf*`Isz*(5F*rKxX)tdN0GV)&;{DqU zA9zq0o*K~NCJ*mVx>E<)Y2YDkS@`qrMTdv0VzKYv%X!S6B&3zpF`(AyUypvO2Ut}> z&xVJ|d5KL&p&T4_jIqDJ!h(ABa8%Z5*qOU0AH0)uzG$gZZ{TrSS+DY5Xehf0ZZk+% zJ@i2{UCnsC#_vCszPH9*FC#8q0kl6%5V7Pj-gw(bN>TCGW{lez(ze!Cwo$+?`8HT_ z;&Ee;2+HdD6kss`g5hKawcP9%toK5WIja}*IdN^dGfGxy9+X5jwT70K)+^WWQD^Dq z6Hsw+u@5OBjC4;v0g@{(eVVko3G#pxuKY$N0{r~1&9>+ebUzklI$#ihM&kouuK+qN zY=;O}NLct=a}XXKSN&Yttq-?lyU;FpBN(-jIqa&(%j@&=^K^_EagD&}kln{?i+_&y zr8W%^Ib0`H>Ssy=eigo+6lsnyO4}LZ&&jpxf5jrDofjdpOFmVje>*7`unM$dG0{ws z)INYNq+odUYG{G$hYVF<(wBS^RHJ0Dee=|58gWoFAiMxyk`5s#C%jc?o9AHTSy3h%x3@~0tx^h!vBeQ&QL4j9YD*XHOqf43F4<9|Mu>F_`vSet! zn(HNi(&nK#T-x)+!stn>%~ohZK~m?E2Q^x`5F{D5l#s1&yKr!zpR>!ydt4?y;ejEvNbIxPtZ)gaAI${^L#*Y`QB z9Z*>vP=ea$b~&l&&0K<B3`9zxw2-z-&FT|Fd*o4y2a>AJ^@}7vK>w$a@JO1eBEn zfC~@!6imSCfc86JbAy@zsaZ@+46I9GfxWIScAyMFD|9_9b(>i2d8A+Gz%xK_0mK?i zS&=R+2=tSmmN-w{Iz4T@wgYTJmCN`ZNXk16md`w~NmJdK2X0xia=;$oVxWU0@ue}R z>46be1-VlL!^(jDJwjD72C1UVOxzPAdcQ^iaR`tE0gZ6%fN=##JAiE3)72%+$%zD} zkPx5Lx^DC<0SOJ7P82tp@ywjQzP>l0ascTN`Sxnr%T#~{O_7_O4c2(HbV?e)KLUK_ zny1;8$?Rvy=*6-@luuBQGzOdv>H(1yFi^~aTLh@+K!B9No}Q7B0WyJr|2riVA4w|= zIL2#$Yz$n2_V#w8;%lcE84nmufc@l-jY6gI7oDn8vvSlVMapnKlkOU|W2iipCRzT{ z^-UGaA|CqJ@nPM8P~CK)7ljt>j~(xM%CJ91Y74SRhFuorKnFR31ijTLNy{&2UmVZQD4fUEHPj;r}Um15twP;Tq(SE zXWgR&#{0=3%RS)T;?m`Yoc6xQ($V1S7%0-x!Mb~OM)m0b_O*7X zc6V3p?(m=TlJGD%FhD>+@X}IZO5bqo`040E-iHE}&fDZ@=l+;2+!&yU4mdDu6hQaV38U}Y8`+wMh_ypbU4UMe; z&O}B4GYeaOlIzZ15+Vx|eiC(dIVL%KQGmIHl&2#=*;8J{*wf0G+k`|=fQZkX=Uadc zz}b+<-NxG1iN~FvJ(eF%fZc zGca>7v9UAL5wS2ab22iqGcs||Gcofpaq+M)6aDvx zc^Dbp+}s%4SQ+db%@~=vxw#pcSQuGY=)WoGojh!v4c+N&ok;&9K@8wz>}X-{Y++|h z^p8YCBRdyoev)rX|Emi&_WvW+*6F{8>3hH!-3{#-nHiY=>C%4)<>dbVNo{QYhuX(9Kco25+|Jp~$=uGK zNK}-FO3u*O!uB7W`dO?>(4jL8DDv+e># z73~I0oKK7k84+EW3kj6SF(M6p@m`F7#vfT&C@=!5%smhoOc4ou5}PS60gP;rf@50# zLglNnBBrqsDr8&t)U5HzIU}d0cKc1Qj3!Bzf?I*_1^?GGK>#H|3*T+sVfcSwD80hJ z_&?CvV=w%_q2UDTzxX#q)Xs6=`!D!jF7`kGj2z{_SCgD-RXlvROc?wgILqyWa1OE- zLqBGdS}j@()s|=9?D~tCs2H4soQIB+qc?UkMa6dy0TmX%PA1(C#>_;0H=jx z*^1&p#v$yZ7)kr56hlm&=h`ph%wS)7Gof#}RJ=c)e zDi27LDj7-+Z>Q-Y$`I_L`z@48WFM*lFo?UxF7tAxoF4(e6hDxAaloncbPftA$q2T_ zA4wc5l)K6zt0Y3Q74eBHb;jbt;YF~qtkVr$my@sSot^BJYfN&Rv18WH86TOpFJxR$Lj0C52LBG0o;&tAaK7p z(I{(W_`yZE0xDzekDe1q+A!TWZt2fDjBu0yQdM^`@CY!IBavlbL;uE`d5`fU7lM}0 zD6sR=>SAL%agt&+h`6kptY9{pz%qQQ`5!XAzk8!blCrhq$=dygVJ8RX#fu1qL&Lp^ zkbL~gJA4{Ls|7JfVAL^V+XJYuhKwO-Cfnf(EHJA1>r!-{A7`+J_Ixy?Q4D}b3W}oZ zIcJr`EaH3l;&_IdaAtyhqPiM?)^hC_o+}hKB+YkzMjwLB5n|M{B&8(UV5f=5dWH9_ zo+<2X1bKuB)=*-Bz$jn>A+r#X6NR79XpD$?Ax|O$e#3UnV0wIjGo;_tJ(|UxRJ0~) z_edtS8(UY9r*^{OBBz2d>&+sdoi8OrLR(Suwjx!A!f`1H`Ug(#Jfd0dxS#+`8+e2l zQS1L6Uwf=t{ta}P(w6xvG%H=nW%^g?1Y>XGCXikC&x>-lYfA+Ph}bSU z%#Z*}ANH3bKK0ANWhR&EZQW9xI4yO&+dI@6IJ1x;E*Dm3zsUf+cCNedR#U!wbF*70 zY)+aUK5tD3_qhz$vA{3;A_>}!bHhqQnTy6)Yu6a1uw-x&NP)WI-oB*pF??+Y{3Myp z&PW}QM5?X;S}sgA=cus)L)F0!1j4a_ypVdsWN_l!O&mmR_%*@BeSPl;FzEdx15l1H zELy02=U7PK?SXo89usQITfGcgwk5n_cyLrYGz}9&LYIbY;7Ad;(X56y%B(Q2Rm-~B zit_k=$9R*NeD({Vpv(p1V;WItW$}&*^h=q+JtO8et`7uHK4p10(To_CyHrwW!ssd{ zgZOT-V&q{6^*8{(&Y@xDmPMf`zmn>sRI}knU2%i#iG5Zymo+FH?`?^3a}gJOG%YP~ zrCAvX1ZGJ+QvjwkX@&FLXh-=ZluEU7*j!cl=cQjmXth3Q)t!U!{&fu(8Hv88JU{z% zL@LRFGh{ZEQBqhrt+h7K5UO4>gm!TW0gGHtc~hA1FLC2Y7{YXrLfrcD-_G)ieJlj? zwX2<-7Xh6(I#N};hLx5_sPVDtN%C!!FOO@Yb>YZ^m&da1={cF@rIY@t!@uiakWuA= zkP_Uk$#$e1P@b=ekC~;A?za|_~x|zX;cTJ+|xUzemXA0%KgO~P5pxoe@9x5s`Y~316sxpcn}VZoiG;*; zM=tbdI?=-2TsPUHn?{75(PxW{Q(R9H;gl7GC;`$I=(R$?6jS{q5QC4Wl_H-KHsWg% zbq+#l3+%dg4bM3-ZYCnGv4Xt?)Nn;H#%eD%6(1j(g0hP(*+^pJNp}piBv7TZu$@@c z>!|%{P4VFA0s3nFb5OfSI2@yi`Sr5famXIYLJ_ROzhx=+wGRw5kuzI)MyDv%{Rm#% z(U0$0!v;TY0)8G|oSwy8e)mkKJdGvC{erE_mf0Q+N|jMN?(an+kDs1=Ox!|RQ#U@H zcndKbH7`eD#;TGrU4uWhwmRB{8dFZ-d%DEWC6}NHt7RDWNE1WXL=}m@ zpe{IrpvCAjvb(!Lo6A*r^mMd7I|y_W-}-RldtYeo=6Jgb$?0$*DHB^6N=vlXX>5_4;lI z3{}-p+bEC#9*gCJP^p}CmZ=VpT&Y=_iGi7!yShhh4N*-7d^gTC(CKU)4to*Se!z1~ z`QfJ7Uc`2%1@URX&iq$ zqsfALD>3@#v()CpgqgT1Dx5d;=V?y~m&LdJGM`^A*^ps7*4jbf1lzsG|31_vhE|KB z*=%oBs2|1MxlXrejRqsWvVveIGfvIBmO?v5RL7#gYWY_%k(4#j&F!jw|#QA}9%D_%3+N+9|O9 z)gnS_O*>)28WphT6xjulS#`!Ib)5x)@7poMxKGn$8YPuwnq>_SDe_1qR7So|0;G)? z4fcEgfGbMuKcu7d?ZI%aTG@`9s~ zHy0E+N@T?S>Qy$ASKrhmkPLu#fzWmE*bK{VP$PRYW~TLq(UPpBD)d359x1>k7?ft) z)9A2(cR6P~hRYp~SiLJ-%YIF)6pZzG#C}I&g$SWv0#qtZBp!k{o7dj$Qy^;hilOM& z&Lv7KNvEPHxvku%6A_cO9#se5b=WD@pz;)Ama#)tWf3(lC0Ezi*(!Uj{nn7*v0vrV zMzUuKFXw0o3A2~sgu-ieLJ-v2!Jd0FORb@8gxrSY3gFRT;+`yV^fD5Ke{&=?X@LI zi7f|%s-uR_#SPM$#FT0*R=#*}F2hZ0Hr1MM8?I?CKJzeXEgExS2XY z80OtTQopveFEHDTOE^BTn8ft~#CvLdRGL0$cMgZ~^vhU-oT!Qn6)%6igLKqZ4P=cx zpoOB5B?<#3h=1C#TO0scq34TXPM!)vSCy$VtD?@+;>^L4T&AXSV0g~-sHRH3jK(nh zix{C;SzTTP!t3SdNnhKSmN^kFD+>-hmPiLxM^J6c6N^$AVq7B)fmV_lqGSp93{NQ(xQg*uBJF0eYkuo;k8{rq;mVJaH-S)yh z1B~SEPJU~AiWZfnJ&oynp|&J~IfUN1_~;{URqF0@{pk~<&$Q>0M~svX%rX0IBgqL?o){G zb%~#bXsXu3B+5wBq`ES@o78z8Q^U=HH${upPo`EImCgekK5Wx7+!SVdQ~{ql+u~z8 z4p`(d=1%w6C4c%LM>&iYgu9r%fu?(&Wtv)5)UU-AM`DnXzgAZcem5m(K~NYnor)Im zUjwUbNg1f258((L6ua$ftH;>;o0V1Niz$rD+up0Fw^NrOtE~}FCNc0(6hN=5SQivM zN5jym@khnfb_pCpC8b!+rpVRxB>$+m9p^kU{|_{f)>xUL*=Uk#Sw{sH{+xhnd<@Up zci?Vm>M2XaBJItykX`)bP^)1Az*7Dgs}?Ndy)^2$*d^8Nr*Ew6_9{%5EfRyoUP7vH z=qll@NVKR!%WOvMxI%C{g{TcB-LIEO)dd8r*cX+U4&$1H!BEyYlgdp;cspB{F*5{X z6G>VtAd!d0Ki0XJyQQRKpR5M|YQYasruC0d$_u$1xt!!q=q{+en0upBVmvrDdpavs zUOGH^)v7#6XVn$IFj%HIbVDZ97ROeEhGC!5IFXc1PN_AfJYP|r$+mH^U^9rA5{Qye zkif5JVL6~lb)o3m!K#-Q(BJtpG>HDI0YVpys0^J=YQYRWU&Y7tOGGir?F&uwwWGOs zZi#cX3$D1n9l=teDRo+FO{^53eM$EslT|YnUy?mBY6tQ_k9S-t_N;|I-Ys>G#$Dpb zxiy~MM|9*T=;+JBq!651Li#ZR8R^7#!l`qcy((?Kn5G4j+THl7W)PY(5aWiE9@L6G z3j2%3RBow~Ms7(pZs{3hM?s#8$LYtko}jDySh`jVQ2zH0j$|L9Yz`c>`@UH~o~I|s zp{9bs{Q|fyWcB1s6hXqdA8lCHrTuU2y-j~g1qEjc_szhomB$%t%TJ=&gO<0tEPDi) zy-^iXmB>(RQRMc}GDG4BP~$IS&EHwLw!n-iiSOYS64mk_qb&lX(42Fl)(OC^OaUOO zOgLs{a8qT!Z2yLYY$Y82ZrvPC_mzI!(6k)$y~FQF?{lSV(wUqaUawZl4w^j;*Vw1E zJHq9Bs(XYbclVt6*{au}s+4657n-z!(-}g}qMC6cWBpFiU>%NrKHmL=ktJ&|&F_rj zcuEhOCl;@v>7>|aD!+DeY%CWb(cc+~_bJ57; z>YUufrj|hvl~&?eWLnJLk@RmSRr_ic-}3d&){-r}Mi2 z-Xb*kt`2Zsw6VgL? z#|Zw!(^&VNj%#A12-MPlL>X349aJXno9?#};u09vezO`c!;@M^K?YQxDU;F|rYw^g z{qbSkadP_O5r1emF7Cy&XMr>k6XrNw#_0M%GL4U9+6t2+{Mj7~Es3u@gL@S_g3%p0fC<;Tk$Rxt1CJ>9$vFZ*DaB{j=@ zm$zC=zE1=!G8X$_uKgmmF|DF;9KEh;14G+to+LUtIxC@M4-jpFa14;tWm?1wv?ki@ zB(?!PfsOZvO@D8$PuxhcdT5~?<3iyaDZaKN*ZYfZt2a~8_LewobH~Jnh}rYBpFNCE z2Tluc+!_g%lLb@{t{wpOH;d8?r)84Y(dniq#)PqKpn6AR3e7~d+f6B>tzj)l1#RFnDDAW~;uahm4!a zHL|SdCCorEG7g!)QC9({{qYGJ`=)T7^Mp;K?Mv5bCVL9nniwGcE?l67%rgqLAPLhl z-tt*SOi;#1T^|&R0;OiR{xBGwdYS#VtjmhBW~p1*+Q~Zd7!#1?%TW&?`<-@1QQ#ag zG-kFlXUwg~Zw9e`em&3LIRnl{U?};*O-nlV0QonB!UuSN6!Lv~0BpT%?gU2-@Z~^h zqQ{qH6H~2~f3Ye5TSv~UDt_I<{)vsbE+^`3J$Kdnb)#?FbI~)puEWgjZ@Z>=YgE+N z4f^$S;}~`sD=l^KTa0@TdMs^5MckGbkG;$MZUiRVn0}YhSgbnYrr$HxOKCNADU{rO z=Ej@-Lg()Q-jYfdNALorWGvUE8LT*>&8<{=rg1x@@%BSkG^k>$dUA7vJR*Z4XGGgIQdi3P+_9jNO$kC@z7I3Oqv~__N$>CII1u7qxshxH> zn#?{4Y8od!4g&73*BGovFIvsSON?qq@WsTVuB;?h2g-M*p_q(ylR68`&vWizd-IA^ zOo1u6P)BL-A_4e=$ZPiT`_T+t8#F*sz7GL&ldIX%_ILEG8pj(lQNBd0*7K}?NNawK zVV&)M=NPQ8wFG!?&ggkrkK0W|F-YnP6s({7^}}!85fUVSz%;B4Bq10(3ZM(d@6jhP z(e?Z+rV7*_D=Ji89}k?kQ+1+~$sP&oxxo@9xU@Eg!!ZwFf2GnB@e8CC&Niy&iJE1{ z?53=RhrxLt9MI#IL0BXx8|CQvW64Ej^IpE3J@3^(Ffi z9|m~3STLHd>`7E`IGriN^DuUpOAe~keXsTedbJtZ{Ur!$>zY^ zFeeQ{NgT?KSJ6txpXNR))5R+d_Ng+}ahk|JPouC<;c|Cyl!N6s2PP+rtna#=*j$xW zRawt)w`-RD_)ak@{;CVWuQcg5_?tDuRq9gai_AeCS~@!1K1yS{oYX_`aI4Dtr0WQd z)OIlyKc=_DBq>_%cwLM5W1yqZEHTg%UCK+s*;;`t!L|sad|q}xtwL32a`JgS!=c;T zJrEVFJ+H+Cuoe@gaqvV|@w1OND8e5&61~wE2pZ80zt=+%Jtz4E9S3#YeWwgAX~giD zwZ~;^CXx9PxlgCwrD^usX-d&;B9+dsFmT!lO0pJ>6-S0%V!t)I{J!762%?v}SZcBU z_N~_@juDD8H@hiUpVSR^f8OxC3KD2@+r)sAg4!`QhQKL>%t#61Fcj6W(n|k8A%iAx ze%yNzCxw*U_MO1VLk>5pJKoL|>|{FM2MSx~l3mIyIg#Y=fE8iHD`Mw-lrk=HtiGB9vsdBiCB zr$Yzs;vfxdd3%8!YM|?tNjcm=^ZxO7qgQyp~#&A|qT~8pN{C(5> zHAB@<*4^4H<}oGRd1(BGFXklG$246ocd|Ab1aF&l%|Qo6P%=m!mnl4^Q6;&h8NS^L z(8do1NnIrB%O9^X7Kj}j*ZJW_=Y_JUW&cd@y~j@t(9dO%5grRrT3bSwc`U9YW25KC zh~YLdyn7#F(maQE@ig>6MJ)=He(Zw?1Cp96fN1}yDg!!HQ`6udWA;tspZtB|5w)E< zf4asE~g8Qv#ccRW0p%e{r~E?K@+4C zN!5{9c4U$H`j@XJZ2RE8n1I6$8ME&Ek{gN61}Q_i&p_v6Lyt1Jnbb=O2ES8xTrVro zmiWEwCyoat+;`f+U92HAed!IvKFRfAEkJuNHG(14+4JG9XcFAjQZO%Mr^qWsSBOBE zzCMZp&znFW2vj1*|5L;HJQ+Rlx5eA@pQ5+b7P%3HDSYOLA!5zb?!vEhXcjn*n!EPW zDqzE(rlyJ7pRn?|`L-&5QVT%E2ouCenP^8E-w<~*Tbb506lu)lS&!0OyvilQz|e9k zOumfm8%y z<8TU`4j5K&@8T+}R~#~(QS@_4VJ+BnTut`RVM7NTNeXD_xo;M;M`9W{QE`}eRr$r3 z-%iqM`Y@7hs;LG$$@T} z)4LY!WY!*RYG-qxrPu9^?)64DEH*9@f8|YQszBBEy!N&Elrh2#)?*}Ts~K&2;husq~Pj%$@rYSE9=*xk5BOWEcOC;U@7=mKvmfq05y&1 zR&(_3#TiiMIYLtOjPyPyF`#bSC0A>PHmRSEA`7?5cZ69#Bv#}o($G2X7ogsoN44$N zvMTcOZXb=IkW^K9Ua{0sf1nVK4~@C2T?Yt+@>0~kN&M-Kj;Hj}`(*<10@A+nS@|cb zJ%E6pdoE8+%fn;oVQkpACeh8B!iCqQ#Tj9RMp$M{z1T&v>eApjNLT(j*Hg>KTsf(# zP~+@Nzp3b5~Uwp z*SazHHFmPO!kvX<9>4llDOC(V>#g6H6D*FT#poW|7_<18pj{(pyS;JuXXRH4J^phA zx}HP5p7Q2h^5JL|_OEDiPJLL2c$b_K)ba@_G+cP>@3I0CX5;eX&>16wY+I~%r(oZz z&krHu>Kc*eB`jC8KMC?JKPtiP?z!9b1m-TiUVMUbb8CTz=Ul`>3ze$;D@?H@6DN!{ zLBHmTyOk`jqBL3kJE4q*QDakLu)$-xS(TIK{0#27YfYaCC-=~-7m@juP<@{9NA`E` zuLxtw1kKPVa4kf2W5w?Yy*D5*s0-GzR$FqWU_{GRNmkpv8b^*zV%x@{>6**RE&{#+ z<;gWkoviPZeu5kyT^})uWO-pa0>wmWLbtNHD)9rWeavgiy868D?pZP3gR`y*k%EZG zDoP*e8-pV+kq_{21*sJG8}fyl8BJ#w|5giAT55oVo9=nH6rdb=i5^#2AGkmMz3dyv z4jw+V+U)KSE}jpC<9dw}%fnM_0^X&VFD@h50uJhpn%PU)qPp@ilKY*gm~V$ua+B70 zvf=m~S63Y8l6{aw>xn`wF*iG8d{SKPvSstLAaLlTU`nhN5Gg7$n9%~nBX!Z}XBM(4Z*M9es2QIwR;n;__| z&Tf~YrIsY<`lHM(cd)|@un;nk$#Bey19-$o<0*|@Z0V+zM2j32@Mi3~5ifRUWbY4? zfyH+dfGLf%EPm(VK?FV3%XAwF%HQ%iSLI!nv<~?7e3|KH=*NzXe)1;QoC-{l8;79{ zabciab!e#>Etv#k11*p?=l*~X1X8L;j1?(XV8OQ$V!QR_C4mE*d?LZbW{R9MIU!L|e8u%0L#(POL`U1tMmoG`9fP=;F0W(h+UQaEOBm@tRN|ry^^qC>elZ~&3 zI;<622+)eP^}|J`U!N{tnQtEh9vLJufmk=v0>W`wQY-_g5>kQb;>hvQAhGkVQ#)Dv z#}ug8{L@p|A9k3aZ|DUpi;LfTpw~x_A&>|ZM^pzL$OJkdhq4$0>M@-JBRhhvM&%AO zU1#WY8Hjw(b$fUqZP@~Ha<86u?}>H0WZy6Ng#Ex0$ajHUGU-5~GMd_QvORHCoI#8Q zGwbX%$+|oN%<$h5jRO+dzc_uO?b`SYxBc~t`RQ%8&^ z%`yC?iONxx%K8>u0uZE%F1BSN#Vx?G!VcUdkff09=`!4&gQ$@s20Ob=8XWK3=?Taa z&56uojB9BLV3jZD2VI7yUWXnD{q% zm#53r_o(8IWP99-WPcQ-1J}v2hl_y-$A$Xgh?DxwO!IzHbb;FYXKofFA;CEFwgH5c;=~j&FJB&DMuOMU~~6|LV+}bLLql zqS$r+D+ED_PEJ^&?TiDl56~R9-MFhypAV<}kWf&W#B)vWwR*k_M;z~Lxow*4KZHpL z5SoO-h!mT>h2&OUwgAtR*h2*|i@g}8h@OY8mE^+>z}yJ#qvmHpU}ZjZc+skmSvb<2 z?R8vH2;`+l{ywfDL#Cv{_I7cFasqS~ zVBGkVs~FjRS|}EtF#{aM0x5Zzf@)OAsvP9^$fiJx^jAP{rF!nWrQE5gTLKHqn7UV9 zd(*8OgKkZB6r(()F|l#7xMfWCtIkQS?wLm=JDebMqsM(@j&F`?v7?UeegGD^ei1Ly zIkFfJ)TuN+hQ`Q+ortP&eb`spKJT=5{g}USuUp&bcC+^VpV|=l%kV^I9$XSL1k+O- zwR@sOk+UYN>?&MAGK_FGY*&{LNHx$%yJ}y@H8TZy7!%XtvB@QxM?k(uoe#-^312px zt}2=E*+Xro7JNq>r=-o_FC@80zgx|h2M15Nw4w6_Rm(O#l#1f2_6nP5^jLr!cj3pu zDLhHwTAxwrwgBgGVtFh~0b((q&WJ{_8n0sDurhC0Tt<@j=l&}k-|g>A5mUd~FSEHy z8Y{gV_4_R)evCsOfeRe!kv0#G>R9H5Smq#7$x^TZN0J<`S8kgT0XncC_=0*IJsscm z^@-HzcEty9cSR}v63q$p9o6E&m!QrqXRDINqYp%s>Vg8({`U$1EOdle_~S4@s$rRZ z?Rq(FC3rHRbmI0zF?HMift`~?(VU8YiO5MneB1luXljLW0KI^fQUg|xOpACzb4m*_ z>s7_KRK83=oG6*U^I6bQ-{joWh-Azj%UX+fI*T+yLx_i$QcV@SZs}A1%f?|2X^69k z?}z9V+Iz@n|i)&BwFzSVH?ALEcV*-)buCcS{*Q zjkyt5f00RZNV_woxKmAK18d473Y^NMR|qccI_US?_HlG@*)MclmacTyx3!yX4%Y$e zt(J}`NFgp$nB&%4?J~!^sjVmTOX`&vO+DcZ6T7u5DOEapPs5UyEd942>CwTW0q;j1 z{9f$Wo1AZdQR7yLl%4!Sc%XC@3jhE|Ik~O|l0~wa+-!8qj945k1o@h>IK(8erwix< z_GCxgL=$t+-3vccYY!#C;mZ2kp}~}Y56t}s{+|PM(xNGX0_k=3{ejI^%dhV}?sUG7 z>?>LB^WheVG8BmmrSjjUg;3C9TrwIxj=U1JgF`oz>HG+>riEq!>m}11p z3UF9lI~PvV|ELXT#wXGpX zS1Smdy3Opsg(jeHLE-uLPe!bt@@Xy@)@4wt10ZdKG zDmrX8Tlr`Blq=Zn>4A*yhNN%Zb#^@3U>ZVQH?h~*a6n(dxPj4BBx zMKR!DO2S-X8Q^AUX=xMjA#BDnmnfiexGv!!z5-oR6G#a9_4c(jN_(9fK0^rwUT+Ew zm_`{gnmtq@CM`AI-`m_EZh-O}`w_ZPrM~1@pF7lKp7D{})KKnV1 zq0~9pPtJ30zfV9@7U`HBj-xM|Z>3GT9eKGe(J*amUbWKzHQPI^PO$9>LckfUS5{O& zz+*wt+Vho*EU^9b;`hvnPfdlicQZn>c5XbHI(NtAM$`@shzsgR#79d%Iqo$$PpQ`T zee#G?A% zdhQXE%nWHMJ%bo7xOLg~^*Y+tHLWkJL#l)AfRQBtY64WMS-%y=Mg`H|ESb#e;m@SK zIA1%>9KJvtxwG>LA}@h+zgso$DJuvFzv|MCr+0fkautreU?PB+LOfZ1efs43J*D4| zD$8>h=j8#R!BDk#dp%vPHBYp+78QZPk)QAZY1Zm>r`dp%(fiscj&t^Op6Qy<)Deq@ zgeA$Vs;c$Wn?MWU*>YmUZaW-j`@BBc*!NYqt==t7wZ4Dm;!d9(+w}D#aM|zlH55<0 z2l&%bzCCbbvP~V5N-)8~OueuEg~rJ=87^YS)NvP{KQN@-*4*t{?66xIb0$&X0z6D{ z<}Q{eR!|9%3%;Hdnv|0$=K{e?=&_w=`DEHJKz7PblaC#Kxj@UPxwlcy3Db@ds)Tk&4Pq?C>Gs>Zwf z`*btU{W9Eupf_)S!tJJVWqsdAbl8W>wPvgJmdhM|uT@$_HMQj`>~u4BjA-B2+Yx)d zF6h!{F8rq)m#04=(MZ1zWl}X)U*8uLcCm650vYl7oXZ|7N!i(XvNIwcyp@xYzq%&V z@DSTKki>YuXe+#T#W8tFF?$2_A4;3TZ!iY-<;hSC&=PK9-@y$D6R`+~h6j6(3>h@v zJ^AD(k-Xn}N22P19#$B1<@s*K`q&H(8x;gM>wQ99V1x`uXYUn7H+pkzwtM{q zsmSsqGb*00HZxCq3u6Ve7F2{zv?o#rV~u(rvfb3e0;MGB4zht)O_U@?pDYUJsPf>w zW$#9`MkQB;R!G=$O+jt6#c9?F&bH9>9W zKL74(w^&10L+Fc@WZk`m>ajR)kj64;zdS@?$0}qSWkV&M-;O16XGgU)S^r@HB4G}@ z?iraA7WVJRUTdc1z1`~%>i+`=9z;GojJsSLdU;dfegC*pKn9Z0nX}wnT)0nG#wjD* zU&vK+=}jima7d!meBD6xa-Gy=@r?L z>h|ud-H)?FNBE6`IWyVqwrcfzTenCE`UceXyKN@ZqN1XPYZ)nyEiI3yvoqN&24cVj z5Ga!6K45rI>QuFqlvGv0*@y^Y^*s)gr{BgV$4!70`#f!JubiBYkMNw+vOB1L9aFB= zViJ+k(()of39*{BfoCNC8o*byfHm}okVetGKpu6}HATg1b*Gc4_4oGmQ#2%AN08(7 zTCO4rh9x01&imbG=O0oFz0#+doSgjoV1N&?bIW^Q?idh?&k@cMGOyYoH+pU2sN^|>Dq@cYG2 zeu&32jW~+m`cJIT(qacfLlv<|O&I72>CVoqlz(-#c59jCEA5c60}-V)6s>tIoYO(Y za3n@(WhF5Dud^|haFsJM6ofBxzAR_E5GxeaF=8f5WeNrPG-X6q5r}cowt4*(-x2w*Yq{E`o_m}%s=sZ4{7@i36zI+R_ z=`>&rxzb;r8^54EJmf4v^>>1gbF)F)fVY%YR364CbN%IwyoB@hoNsrIb9^usXEJ5B z{_b=?8wRg7Wa0f(nLf{Qx@uPtcp;07-t>Cj?0Y7cCJFIftxNKUwg1X7%XxV*%N^ha zHSaN=Ok2^@w-%@J9i#OEGd_>ou66!Fpk==Eatx|$`t)!XuP&d(R(`xKZ3D8Pv@_n# z_Emg*6n8|~MUR2+ddg&4-J|`yS)<$8(AX$_lCNGX6QZlM!r5xMbeE-P7M`xegq6j$ z%WMW=Itw0@lg$cQ1BcPr(GjtDGdi{~bCRv3=p+(~x`Pk208M$d(Vj?_!ySV)WV>lO z>_7Jx8nwgilIFWWf!0s=bm_xTEx(<`xQFS<`5x8ndx4Z+*y_vO1I%T}czBDvY&Lg+Azelb*R z*E4qKZvbE0adZ~s?UZR8dpTP__;ktJ*I~2Sa5%x(1Ey^nSOQvBwzp^0N1M$EyqLdM z>(`5B1I9V~+|^Ut_3lW7WHM^%KRqVLnrvFRT&?G0vwLGW*k{|YbC#&UCtMo~cV}iv zVUIIS=-mH6nR~OReX7wyXRLgNz&wl1i~?=7YDZ}n+Uj?kKrR}B5G`J4aI~Wf$z-$E zWO13%e3N>HA|PDZDG}8LvKA%YU0GR)+4V|e)!tPir@;TX7<3FLvFVJ;M6ZNLER9Nl zUz#aEt+Sh(s*6$DSqAjH_ZwSHGuSAXVnw_R{>n%2{tgObj zf6+;6Z@xB|0)1W|f}hl=2oWVMVPIkD5!g-!V7K%>Q*sfYpcsS_ySk1}qf}CrJDtDQ zPj5{9C&sp-uWS*S#d+MWx(wzDqH#HUS{+p#q#ZOhJxGut2?w$BfXx=9FEWlezx_wL zs?AkZy+88Fj4_6bIFKml05|Y|U%7E#y&L)$|Aya5NX|d_|1%l+_x8@q<-r#i7LcjN Wkvu)p=jJ~*@1(`$#cD(h0{=gHoFJJ1 literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/huawei/huawei_logo.png b/content/ko/case-studies/huawei/huawei_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..94361a27eb5af3a06992dda4aa1b411ab702ee2e GIT binary patch literal 14274 zcmaKT19T=`vu&d`KN(#F8dMA5{+*u!DWga-%+l+;35-C12mn#;(}hR)z089H|x`)_R^ARc~q zdjlgY6K5hr6Eh22UXtt19ugu8V_p(9Rv88vdtnoE3kgq06D3bsWg|~3BTi!yem)`| zcdl;$8xv;(B6k~WTPH4eUXuU7<@&z<=QceF(SL|ITk(?or&H=O@t%&h}jN^lol$bZ*Rac8+HBjGUaD^bAb&OiZ-j60}Ypw$29bw6;#9|3VNkaWZnW zuy?kwvnBcm(ZJBo#hI7n+tdH-f{p!u!`eFi*D!q#7`?lJJv}2G!#`d6m!OQy|6kO` z=D($#oE1&}$KU@au#>Wfy$QXdiIbg+qtSQZOiBMSWzQw-Xky@O=csIFXZ^2HlsC6? zwsSJKvnLW3CIZM97+Ki<)28|l1sNGGNn0mp16v~#NfBO>ZwNXI3u7)3F$NAXP7y&d zQ889VMp1T7Mov)5AAHx!9Q4I{!=8_3`k&rq4f@|IOR) z#(y*4#P*x+j^Dhlm%`}<0%Bp66cJQ*U%%9W^hXn2zB*^HV5aF5K?dg$L`FoG9TXNt zLKp-J&n7Vie|Q(=omPSiCn6FQ1yknf0}2F$gB(|5$xDJH8ld8uQMy)ssi=&sX$&gy zV02n9=jLAJ=APB{Jawzp=@u9>jd@I2eJu=o@dvE%pm|PN5y1RDE||5_y^}v}CM}HE zmXo&|z#2Kw6b(^TC9(TZ007`}Hcl{e1U4GT+ijW_OKEq?q4P3g-hmN_uiYNx-JxVw zmVSHf@A%eO>(XDpN>)~@T9HKnu+U(&B5T&e zNAT!bEc^fFqBr8+?uXDFT5l=d^*cpeuO#L{I==EjBc&g+cn~MpTRV|%iI$d>|wnTbRiMhy_GaLOtSFCO~{PX4yXbP?`& zVMr>uUH|Czb$36ambiCFjN{97F8Dg-B4JZrIMDNTl^`(q%U4sHC?F36fR#XV53v7X zc;QP(2v_;$x=5Eg|>E%^Ac3jxg6YV59@*2M5usK_}8PQ25&I&9J} z=t9?gb!zGlY#aPID(+SiNk_qW%ygf!C<#3f`bcjgBpi9;;QNKH)PTO%jb(ngMyhMVPrF;@e*efhK45;#pw!?9*ZL zva~HE+zAwU+1cbo?m_eA{AYGF>SP9LyV4!-Uc^%K zOL_Y*DFcS#>;T&!gX_RF7)6Wj-@q;l;or(_Zeg&a>ACpa)1jOf!W<`FPm+ET= zq#7CI4XvNS0^^^rq`nUZB>jO4M2p^a{?X~zAJO(51Ry9YE0g-HVykI=5^lHyRjSqY@teU@ z8MWqpAB(OjbvRjB{-CQFsJH@|g@m7T`Xq||VMhdM$@!8Chtjn!dQyl8Ctq1=nS zDk^b{SS^@ZEarjH2iDexxRI+7()d*+7Q0c|3#*Jdaj9 zos;cGeCpTuY=zbpDT;{BC$=r@OkNSjH92oUhwN+qjvl(t0V`t2R})0$ufu7D(Gw$V z4daCO@ooGixoM77g-S6m83xB-6X{NC!&qh`_hT3~T%!04)1a5jg&!O^#FoMXp-E=- z`4v$aEDF%i9H_CA_RKx;mo(IdO{tLW&xe?KVtm>IFYlKI{ug_o1(og0OZXeSXrFeP zN53M-Y6Uw5Xk-FF3Nq27Q1ZN$uid7n&>eA+lkG7+<84A^kOQaOnOe#dhV&Sg2qOcx zibJd_snQvY1RHIE=2-eFYjMv-npDdbC}5k`$tH#g%U93{0af0XIEdV*Y+6n%e;Y18 zxf^J_0PnxE-lxnVUl;4OKYP|E-_CXxi%M_BUn~prqf&0 zTMs4&OAthEN3=+H}{$yF|Jmy)CYOu#NPSY@%?g_X@aMV;!R+y#ubV9$viy9hD%ac_eL0QV;+K5?S;#DmjoH(VdQN6ZcD-|4b=dsOXViGxGkv(AdpmMGkz zjaPW+$Fp(EylGrL$7-nOoQ*^7EQPL3W$F{l4Y)cDY0?-O72S0Eug{?5L< zn$JfeVqAP|jd$_TB5IT1+o;9!4Xvfnn-Yo6lYa-zH47oMsA_0{hmk89PLVE3H^ebH z4W68OH&Rj#zg+zy3~Xe?XOqV=L-?a$T>?c#kzq6cVrRH8gj?nMCIS#2-A|8I&aS%f zaf(Iw{2uO{8VcP3JY6z%6%k!MD89B%`6!?$jyS4TK&U9>&5ge-RWo~IdX&s$)wX_Bcdeah-0jM50O(r2n^%#LL(?I!wD#|+kpjJYVIRjuX6X|}7WI*2i{ zNK2gfzQ3@lme-5PXcW0xD4dl`i#`Ku`B(R7M1}wc(N4{boulB^m~V?CGEdOxkoAOA?m#Ewj_F$#52Hslf*QcYlBW zTkF*h5jPqZlYoMlA@DV+4qGq;^b|ys@R3$GBl$W{>&6VA(3(v=W-UnOoj!N6Ndd~l z;?;FI#qR2FUM=E{7>?0ia0*H{cRkL!(8;LK-q@HI&)iZ zL6_+H;IHV@Km@C^_^KoWxLPiOm?AW&=^o|XSKI14V60%$3fFG6SoZPvXEnHjk3HZ7nOxOPZdx@8UU&$v8FKdbg1a{K<4$_2`;!! zEhVPyY-7PaI^qTSvlda*rM%K~0UW-W_ymQKU>kGTca@ue-5AElx3|8DgbSmxbPJv{ z6y26w34<_oIjXkHsX~SOhG?eEZuJZmPrsUQ8ls45Gs)mm$wyA#Y)Q(UH#0tW;IJ$> zxd{D-iCAFKC8bwE1pt@|={{WTpfN-2oUe!-n3+;>wI#)jh7U_@nybt34LmF6s@-(I zaZfnh@C5K^r%_9)Vt|~4qji8sYv>H$qS1a(zo2UQmh0RH`kV+ATmlPI;gcMyP9U%- zK^TgR2C~ptL9h+RNWhO>D5$vgWFI>t$a*6?ds}956uRuq zGmT4S>XO^KoOAfw_5y~l+20b@|Min(K&N1SP@=^V0ado9Zr6Rq7xb=gvhu|AY)F(1 z9|2!>Df|kH%?xyfCkQ#)Xze{+0qvLi<|8aNvxO;J5OJ8z#Dyz-OahR)tf{VZg^y2! z$WYNiUlglyEO`w42oU#h#i^K3%tT!NN>zoBpb}z8SXG2f7Pch03w~e6WSEW?Txigi z$lu;z2Mo5C&CP0GOi-L`%V7x>hgI`o0afEz&M41$fr47V>K^MnjWBo04w(cye&9>0 zNree+MGk&`g^xk^uo%CED<%BY*it>S`cz}1&P!YDX+@>P$gh8hM91C(?R0>&Kkw$G29jO~tq6-9~U-sRHRr!(`#sfKtGBENytbPcP$nl$w|T$^ft`buh5 z+*-RA(-Y(6hBdemLI=*i0##FzITOD?Irdji3t+s{FOtS@X+6hoRhLK~-!u`u93sZ3|os;Ol-tEVqL z3EkXf_qseHafu(U?Qxmi27u#F9ZE^*atKIb!r7MBE&ZK`#*a2qdFUm53ZfkzX5eXL zE`%+0H*-F{Rmzz`Z#m7=RzL2I!by*?vImF+2pnu&4>|U0ERQBk_{-z_%>ibQ=|+Y? zI3b5W78)no*|%X}aFVte7Q(g2MdEGV;!)EJ=Za)tFbqL(ern>FJ)vba;BL(hL12zM z7Hy0;md;mZ@Qc%K$?*xZE8*e6%9uGPN7yg?YT7Kz4QK5*$LLld{pRa$@R?KW^JS@4 zl&giMRLKIE%OA_8^Tnr9*6$RJh8gJGlbu(p+0y#cI`RnZZL@()#FW%^LTL3F&HB-O z%)P~HC5h?kz*xgY8a@yaBLrq(Q0#Wq8jb#G@5@MvZ-8ig;UZcLX=X$0g;?$ii-}3% zG}boUbq3~7=nu1!!92@)=D^pQsV}55OzrY$jLgldEDEg`#GIJ>+$4P8Icow#jmMGy8j(S%b|WH4wY279!vUe; z=<0KLmkv`leLAA>drpB3%gB%+^ocB(0Ur)v~i4jYq#&`7Qc z*Uq9kD2uynV51;#BrzclBaT^9i>;O`1=U1bk-8dDIFlQ$`OqKU-L5($q|{9ec`@u; z$j`=#I!(KCyZVq$<42mM!DyE4=86X!#oC0s-9d>GZ`KsL?Z~DQRxgJww)lMN9MBSV zfc~)99c?FXYR0|36X(+?dN-eZK)Uo`tGJI}Z;C#TCj|z{OR-_lltczevILNcWAL?f zY{8YutEE)K(lc(hjgOD7#gXXwmzWZZr}pN4J(`SG3*@u%sc z#&)+0#fkBv5m+V-?8%L6zi!9-dZp1UpUbVnrbOe4Z-_8-C@khnJgSxGy?aBJlRsqIPs5R;VfGbcXP~TCp2<3O zI#aRpBycq-u63ieZ9$Uj_x7O?6t?sxG2wXmyEV%0{VhQJvj+Y8N#4Djw#%qftPuMa zQHk)`ad37#j(_nYqmJI*^T||>X9hA*by#gHvbUCNA@)ndE3iK$%2iyg#@NpEzDMAP zb)P^)`>Qk66gCvN%n%IK)ufdbN)My-y&rx2_;!wok!>O$A3fi)2y&-2&B}h*dM&#$C zkK4*Ad{JGD&1QT{Zi>5gR)2*Yac-GUAK(wIv}BM=N2oQpz4pr@i!?k2U}v~>3*Pw@ zhSxBykysE<=cLIA&L1p@#_|B}v>0F$KO9rr4c;KkQuChhmN1iNc-qiO$U|;CHu{0vIK&mt{iM!60V2MA5wy zSTL&=o7cV4i6>cK_3dWV73^o)7S7t2CMl+Ajxcpalr5Q*Q2;(C=LcPzq>Q(dNHo7ftSlFKbw zORF*`^mr`@&JkB^IZ>u7J3qyyj5|=jGli3Yf%Dw0J+ou99WJKN0ky83nNsvZAzj7D zM-KpxaSsE58jA{Xk&BBwq46piy?J7f@lqpU9d^gp72~a(mJ$MTCK^k(nRikHszBs< zWwds{Z)L;5+G<)*p5oErNKvtQOriGH?e9flGK=DgMla^#R6@__^+^qvrY*b`oaWpj z#S`CTKY&9~i_eHhazg)J)r<*dnll=>^NrXApUnGe#}1Hw8|*A}j^tTWTx_@xD!PJL z<#EKR=pvcSak8e!<@Fu%F7iS~eBzirL?dujhAA5qQJrMV?ho&$DN<@P*}2Hb&7Dxh z+=b4UYVht<8;R+ZrGv=%w)aPPan%s3evG}6C>qiYW(}oh1Au`%cqdn*#7p$WcF0P1=#P--BQ}I0`+IfWsTz6IrB=@VFRpkGjtuM z_Lm2&H$3%!*Ov$Uk;2+YTk#73hdz9wq`%?80;_50K}mGxz!Z*N##5vsZQH6{$0j^o zTq?7kpPAd!bc0gJmZn!};7*vPQsa(9FmA#7sWB25Ri$6=FVRnDHOiKllpvoI`U8CV z7HCR{!U4=_PR4&ZM4G3~jLqV@mJdDJkJl)7gDwe##4Uw>9AyfS2jq&-G@0cc-|0T) zI95g_B{?28mkg=F*LocBlf5Y3?s08N#zwQo!6LSUJ36B4fWA{D!>ZuiES*ZFye~;o zW*i1RRC+8#)e+{ZzlBUVzkaN#>FGU>CK>ty)4pX_rYlsgHakH7D6^bCSeBXSxzm+Z zOWMODz8Xa?B|fMG zYZ&XHWPM%nQEkg=K76|PDXycz=@m&wDG}Pj>5zjj!R%ZbhPN7`@27J{UzvJ!(w6wN9J z64WX73Wu5>ii08JrrCDIds>PNFtza{MN3tY$ywrjIR%r@3#=%!Rw&n6(3;1nl9EmO z@eeK0oi8P-Fpzqw76H_yZsnN}>zg3YDDHj?k1ua;b-1d`ZmCyT7gUEwM{#+Zhq+_k z4CRzbEUTgQosc0;xFSrZ%C{=h?t>&N)qAAD@21(M?IhbB!zXO6TxMi+HgRJ5;D%#c zcJDK9G$M=LUGd2;LgiQCw6OlE1QfacwMV)RMt#woLIp7dyf`&J7J|_3q;}muDcmrI z78Al;_HR?;+>K+!IfWF`H;y?N9D1Q9F_>?#Ju|vl4x4J3ab@PU$e?W{r{+V3NFl{{ zya#^NMTs!3-qlUz>Snua3*4Z&kXx8B+fOfoiH9ZwhS+PRd|RiTF?_(EWPgRGV~tp3 zf{oh8qn<(p(}Wvg5KZE`l$i)O-`-}f=!$d`IgT()RS4#2s)5rp+@JKYc%2VjPKax= z;?`FVZ~L`6-;aJ5_kD;1Zy(zDqXo&udXx(nmb;5|-z%SH6AX;wW-f#(5p9O-3}Rdz z#^1tArs9Q0SS5Wib4Tt&lVLyYx_`7^;lf50>)B5BWXk5^pU;DC(rCINEy0^}A13rz zE~7;ul%R4Of7t*g$eAH!{z}-J)IG1aG>u~`-cdqwxlG|Pj23ogmJv7|QLV3UB*cg* zA|`bkZGifn@N;NbyF^>~tC&p0p?`H3dC&JOfs%Jw1k(B_v1i9^If&opzX}^>9emH% zw8DS7C)$)ySQGeuvM{i$2;h(z+#QTjWXQhRpT?!0BVodzujhEaU6(y~4Ke}Cvtv}R6 zP@q9%*})ixJ{@g2!Rf4ru}^L^G`^qY3RKjz$di4MB4Pj!& z-Fc-)>VgQVXrvEO6Cq3#Atj?5X?jE4%VK4p%?GG7nPJ_{ccOP7OMsxLq+IsE+rAz; zeJ*rrU6m3bsLf<5@Vx$fDT_h#OI%8czCe(&&pJu2$m<(7)3FbZ1>$tk~Dq%(lvXAI$Z9ILQuuP-tH=UDeo6_(joO`cGYkbue zf%sJwNlQh8i2RoWJBI6&p=igyH%}ILCJ`tMv%!l=k zN_am)vq2)m(rO)Vxte{?+8_9x@tfc>G8O(%vC2#R9;-BkIACekCi?(k#DVdbAKqNK#3 z%U4xU;B~SbOTRe6&ysAx+OLTwNFE0L1&Oh2v+R1q6{s_Pljr)1>{X=DnA`Q6wAe!O zrTz0nY&;S?qD=&3-#a|b{qVp;a(7J6A6ZV-yIz4ow;ur|!@K3G-WB7TD24e>?2Ea` zP0wuar+(>br4Q?+Igl;6yK0$dpC7vZUQO4af$rs<|iIrM8{OO`j>RV*Iinu;+v?F%ICp+xO zlD^Y8&iz;H<~i!Wb_2w1pc2V?rXp-E*6=JA!KJ~t-RxdZyDpK{xQWX#Fw=Iv=yxsV zALhxqE(8Kl;hh;buiC2Nz;+u5{yL;|CCVcwfRDVbcgeT+~ zf3m6~6$m3Q3qmPn25-5eNDgOn#{p~g8K_JUFmSv@T0L)SpHTkbw}GY4Ce(UEUPdEB zQQCNaF-ffww{P-FbC~S%CmReW-aoe%(;wqxtKG* zuVJsu^U015ViUV3PIk4oT1VEsG8~#);ne`V5H0M9I> zM45;Rcfo)I{lg63AmTu!D9>a>OH@XBMAt1<W!$(S4b)7%k?o-_IM~Bz9LH+W>tv@PH?dPK+B#?uJ z;PI8t;8|fX)lTd*kqE^0$1BV=8hhC7lcSTmf**t&!B(FyI7O(KiLPxD{ERYSbRjiKb#|AsUE8F*s{i(#o zp1iT%D=z^;ICt7%miOsBEuHA`2UfmZVQjK(wov$0jfE}IN93Fa=1jQ^6}Th&kh$^h zx{ZFe#tVk(69Tix!ssYk@-{bbv@23Ovt0x0$zWvEGI-u#@$pldJ`v%EfAeJ1+mSAJ zr9n!ig*(8w#`Wvdv)A%a zT4Kf$lBK#^Q$b;Tv$@4t?hOCd?J=&&j$hDWTmBhv&kUtxA`w*OxYdscBE?K&fHB_K za)k@T+aISf5NjYHwsW%jODpFCeC!X1_aBOb91V|Aw-t!zdp|R$ZlZ`3TUA>*sZvT4 z$6S1Xr zKrX@Nk;M{q-8<-JnQX>Cf3pZ?ngWE9-lsWWB@^OA@2-7`$Ybv&=;$-5Gr1I^#K%Mw zMm4~Lakhi@v&dw)l&}O-CCSV{z(EuzQ~}f@E4924+_&CA-9g{Be=pDCUVnh;zU8t1 z1-$W@`dG*$nE-q9<*uMW(`GF0JMdyC6(tB56c9;j+SF(&6@?Ax+dtU5jCQsUP+Q9u1kndigLVro}^)<)Y@)M zgFBg+K4@wRxui+XpfxrwHz6;6mjEQeg1PtPanu!-j~oar1+c6F5f%)QtbK&Na(zH3us4WO?=+04O#5XQW5Mv0t)YU)?>XgX zHe990VuK!!=7yh$B;0#Sp=eW3aS($)%R|7JLKVRAF@uIM?O6oe!EyraKPVxWp(3cUSyfw7_@K;o>8$b7*YRSU0m z>CPFwu!_N&$=>OU71(Z$DWhVh!?kEIv$sX?njnR+ldG3nwvPky?bJM%?hT8eB;Ju@J z-}_DC31@_2PEnWjI)b71>=k#@TY@t35{t@^_6B*Chg)gf_XF1rlA$4|nKL@zx>*iz zwuMtATg(}3j-~*Kw|fzsZc}7D7;$|emEPxP?qSv|{+s$?zC=AAjmKRbvMGhzCEBx$ zmbhT4v=~9thJ2-@1kJdF^@CD=)K{mpP!ZxS>e$F#f7_WqDIWTZ=aI|OjR=p*0>^Ws^7q9>PAQAfvTCVPJ#yutCDm1YSamrw1qpDn30 z)6@o<;sMaO~#u|&Q8u%+}CG^pX4~$Za5r-=-2OU_Z1ai8*gM$ zt3|g<-h;aDYDCE^xXY`Yrw%A72-Rtx}_*a7QI)`w-BWEKDAP}<0wp`O~v&Vcmb3!=xoecWljxKW5J9gdt|=cgC}&u7WOL@aFek^)e)kw6n|I z8(Zmn=x%bf;$-UAdg>vQ;4z2PUFtH>QE6G3z0l>dVr9!=&lr}w3XTDuhRVVv-m$2-tf#CPM#ra|Uc7{(r2|5;Ybki6Q290In zdaX+1x_gZ>JAAOH?CGI=s$veETvL~BWqG*jhj7f@$C^d~1$cSE!|(Bwl-q#$(Olku zcM3&a(s}6Zc$HB?gb-aTfvJhh+srVbI*%#gg8ysR-1mMA-Mh@S*je5Dp(u=+567CB z>!^|a6>YCSl9YMLuC2yQoKCONm6bo#pWZ%NcNi(M9udSz65d9(Aj2jk+qIr-g+mc! zyxd6=(z#527Pq9HfNLaN`|SoydU8Fv{@Hw4jm8d;hqCH8Od8(yaT{gqmsokAB=GY7 z`uqq4knKkRgF`v%D+I%k?QUVXlSoH5~`i)L`_?Wxm@!-9B+ zWZAVJt+I`9xrGlI+DpzpUT!OCYZsSirkEjosi8MpwJty+V)wi1z8$62(dodH&9l~+ zRH@pteZPHJoz{8NCf920r9$CuKogm-EXqnZiW$hpHjZ~;c3rXl;!#Mfn%;hgq#5X zg~g;9og9@{aoOsy^U8HeRW3{B6fiqKr26Zwz3agt$AfX<_4(rW=V7HIugz^H*Td-q zEQq*Q20(hhD|}GXx_h&ttMj;cxd!C6!oG`8nMS(Rh41`{))@)n1i^D-C2V29d`gV8qnDJ#I98o->UPiHiC=?(#FV0Ucq`YBl~rEF2oX_@}D2($i**$sKFg(dh4Uj@#B!ncYqiXdpajNcZc9P|w%NIPhl| zIM*ryBm$m#gLeo&Qv8u|9^VPDmky&LSkn$rxHyIE{A#jM>6^=*C?NyRMkvY zE7$4v6W`~)w;Leq&9=W6b=_CXzq6XZ$b~~EbC}b32>5oYq|&xoJd@2$MS9+*xj|x5 zSf~J}vke1lC%JdqUe_UH6_t&S6U(%IpLcn*n*$aBqr}le@&Hmw_L1+`%nQ4wa=G`TMM|YGLazp<_N~1`^Y>qx{ra8pb)8F# zi!|P!9zBHN@>ed8YvADE(kVME=M?hje1t?(>_)Hmn0mfro;>T#Rx2$w8@yhR^Zi*6!#C=lS|C1 zi+dN2HhT^pr&njB%3gs#avTMsQ5g1qdqJbcvA$okQem?0r^KcedD*NNb7u?Hpna^~ z&gaUqlrE>L)#||U-o|C zB-!o0IMdPMfcOb&qFD^dKlIlo4pVe2a~q>QP?jv+@3O(hcgSwPV0H`T@AIHIMb1tg zVZB7;7MK7Kxi*)(y@CQ`<$+8l8uYMc-Sy#a$(ed9V8!D8i}|ZPG7?6BKf9!{uuw0r z8%U6dUuGrouLyt?l!i$#Wv2DKrgd1(eH4a<-DbI3ugBM}OOR!1KTWUOXEHrHI=ZBe zj(o(@@?<&(m)&;q5iA54B1Z-ju?rSJLrq&#TMI!+Mi8gxvZOM7F*Z4FTww10S7f4t`V=X`c2q)1SLo0Lt{gcTNuZcb8&+IJMWw-sBsZ*8*YA(ZiA5Mf zC)y=zHpbjc9atsxHJ}tM4;*4YjfaVjm<=F+m|+mb3$ZFY>`C76Hjl!ZJq)kj!Co z`C;GUqFLWB;BFsG6mGR#{jwiLUmi~hbb1CQ4=N9VfCrTx8dY6g{{EO{_x)`EJQ^(b z6GxMHuGc!A{!gV?B6kV4>YbYnF;(8E_YKkNDahXs4K=`!qM~oWkezPVFISz{9g21| zCA8XY{`f$_%I=el)oCCC3|`AL^<`WsP^c6#<1=>JPsqsbduwGU!2P}a)O2*;B}AP6 z$!e#-JfhbKHU?I$@;&j)c$>Ib}TJMb`KiDdhx zaJZ=`sUWNichyaNABRTn9Y9ZRvDkOxRZ=w^(|&jM-@8U>^VXaxAg+#jVd;Q&@-*(O z#B^Cb#*=gX$UW$yUj&A`>u*BLj*;_`zL+rZUDp^)se81ZwrX`c8=IOW{Y%yAq{4JG zcPPJWh5b!kv#1P32CQtJGh{Qr^?X=JUS2zbZ0t)H85!0wyrd+v>@{xkxQjsq%KinA zP+Ww~b|*?nnOzRU*iM%!R3C9vN}2QR<|?umFa@xp&DJLmmXkk+fq@`?y>b&juzNCN z8J}aPve@m2l--L9k?>!IwPvolUQ@GnJWl?|;IacV2R!1r9K^)nu!rJ;+VZLIeY`yd z*@<`RN&srLTCYfJyAtu+-XB&pzUWu?tLlCq6Xk6lw;gafU)KA)A=H19#`@*GX{IkR z^X44rmzts?$Y?3^2hdRE$Lo`n0Y{sE_sNsb*$^gZ0Ff62*1gl+ymaMOqsl~@o(7Ni zDntD`P4&mKQQX`e>3}5b8iMgGco1Fo%d?Sg*Nqx$8}_R81Ek?;wOm9nYkRO#2v8A? zPK)*EY(axLS~SfKWUuW)sf<;RHmn~N|6k}#9}dIA6ni&~84($oslqfp(i8Bg~>+j9qwfgSoD9@(8jcyE!{p+24ssPEL+a&dfjt zx%+;7{#kEsLeTgwg>5!krY0w&hYr+YyXL=~N_5`t&tOO8%lo0kEu$hMY4mJnMKQbE zZ{hiA=V1y<9ejP8oGBV7EOa`)j^Nxw1#Fwj$~(32%YIGgaMNox8XpYDTvzKVDQYU} z>Ly4G3bJwzs1yClK`>7Wq`>EL1+zlzYaIx$<)-e7**B-7SzrHZ_fvf_K W@eBhBOaJ+*PEu4>q*h2j@P7e3cUl|( literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/huawei/index.html b/content/ko/case-studies/huawei/index.html new file mode 100644 index 0000000000..29de86f5c4 --- /dev/null +++ b/content/ko/case-studies/huawei/index.html @@ -0,0 +1,101 @@ +--- +title: Huawei Case Study + +case_study_styles: true +cid: caseStudies +css: /css/style_huawei.css +--- +
+

CASE STUDY:
Embracing Cloud Native as a User – and a Vendor

+ +
+ +
+ Company  Huawei     Location  Shenzhen, China     Industry  Telecommunications Equipment +
+ +
+ +
+ +
+
+

Challenge

+ A multinational company that’s the largest telecommunications equipment manufacturer in the world, Huawei has more than 180,000 employees. In order to support its fast business development around the globe, Huawei has eight data centers for its internal I.T. department, which have been running 800+ applications in 100K+ VMs to serve these 180,000 users. With the rapid increase of new applications, the cost and efficiency of management and deployment of VM-based apps all became critical challenges for business agility. "It’s very much a distributed system so we found that managing all of the tasks in a more consistent way is always a challenge," says Peixin Hou, the company’s Chief Software Architect and Community Director for Open Source. "We wanted to move into a more agile and decent practice." +
+ +
+

Solution

+ After deciding to use container technology, Huawei began moving the internal I.T. department’s applications to run on Kubernetes. So far, about 30 percent of these applications have been transferred to cloud native. +
+
+

Impact

+ "By the end of 2016, Huawei’s internal I.T. department managed more than 4,000 nodes with tens of thousands containers using a Kubernetes-based Platform as a Service (PaaS) solution," says Hou. "The global deployment cycles decreased from a week to minutes, and the efficiency of application delivery has been improved 10 fold." For the bottom line, he says, "We also see significant operating expense spending cut, in some circumstances 20-30 percent, which we think is very helpful for our business." Given the results Huawei has had internally – and the demand it is seeing externally – the company has also built the technologies into FusionStage™, the PaaS solution it offers its customers. +
+ +
+ +
+ +
+
+ "If you’re a vendor, in order to convince your customer, you should use it yourself. Luckily because Huawei has a lot of employees, we can demonstrate the scale of cloud we can build using this technology."

- Peixin Hou, chief software architect and community director for open source +
+
+ +
+ +
+ Huawei’s Kubernetes journey began with one developer. + Over two years ago, one of the engineers employed by the networking and telecommunications giant became interested in Kubernetes, the technology for managing application containers across clusters of hosts, and started contributing to its open source community. As the technology developed and the community grew, he kept telling his managers about it.

+ And as fate would have it, at the same time, Huawei was looking for a better orchestration system for its internal enterprise I.T. department, which supports every business flow processing. "We have more than 180,000 employees worldwide, and a complicated internal procedure, so probably every week this department needs to develop some new applications," says Peixin Hou, Huawei’s Chief Software Architect and Community Director for Open Source. "Very often our I.T. departments need to launch tens of thousands of containers, with tasks running across thousands of nodes across the world. It’s very much a distributed system, so we found that managing all of the tasks in a more consistent way is always a challenge."

+ In the past, Huawei had used virtual machines to encapsulate applications, but "every time when we start a VM," Hou says, "whether because it’s a new service or because it was a service that was shut down because of some abnormal node functioning, it takes a lot of time." Huawei turned to containerization, so the timing was right to try Kubernetes. It took a year to adopt that engineer’s suggestion – the process "is not overnight," says Hou – but once in use, he says, "Kubernetes basically solved most of our problems. Before, the time of deployment took about a week, now it only takes minutes. The developers are happy. That department is also quite happy."

+ Hou sees great benefits to the company that come with using this technology: "Kubernetes brings agility, scale-out capability, and DevOps practice to the cloud-based applications," he says. "It provides us with the ability to customize the scheduling architecture, which makes possible the affinity between container tasks that gives greater efficiency. It supports multiple container formats. It has extensive support for various container networking solutions and container storage." +
+
+ +
+
+ "Kubernetes basically solved most of our problems. Before, the time of deployment took about a week, now it only takes minutes. The developers are happy. That department is also quite happy." +
+
+ +
+
+ And not least of all, there’s an impact on the bottom line. Says Hou: "We also see significant operating expense spending cut in some circumstances 20-30 percent, which is very helpful for our business."

+ Pleased with those initial results, and seeing a demand for cloud native technologies from its customers, Huawei doubled down on Kubernetes. In the spring of 2016, the company became not only a user but also a vendor.

+ "We built the Kubernetes technologies into our solutions," says Hou, referring to Huawei’s FusionStage™ PaaS offering. "Our customers, from very big telecommunications operators to banks, love the idea of cloud native. They like Kubernetes technology. But they need to spend a lot of time to decompose their applications to turn them into microservice architecture, and as a solution provider, we help them. We’ve started to work with some Chinese banks, and we see a lot of interest from our customers like China Mobile and Deutsche Telekom."

+ "If you’re just a user, you’re just a user," adds Hou. "But if you’re a vendor, in order to even convince your customers, you should use it yourself. Luckily because Huawei has a lot of employees, we can demonstrate the scale of cloud we can build using this technology. We provide customer wisdom." While Huawei has its own private cloud, many of its customers run cross-cloud applications using Huawei’s solutions. It’s a big selling point that most of the public cloud providers now support Kubernetes. "This makes the cross-cloud transition much easier than with other solutions," says Hou.

+
+
+ +
+
+ "Our customers, from very big telecommunications operators to banks, love the idea of cloud native. They like Kubernetes technology. But they need to spend a lot of time to decompose their applications to turn them into microservice architecture, and as a solution provider, we help them." +
+
+ +
+
+ Within Huawei itself, once his team completes the transition of the internal business procedure department to Kubernetes, Hou is looking to convince more departments to move over to the cloud native development cycle and practice. "We have a lot of software developers, so we will provide them with our platform as a service solution, our own product," he says. "We would like to see significant cuts in their iteration cycle."

+ Having overseen the initial move to Kubernetes at Huawei, Hou has advice for other companies considering the technology: "When you start to design the architecture of your application, think about cloud native, think about microservice architecture from the beginning," he says. "I think you will benefit from that."

+ But if you already have legacy applications, "start from some microservice-friendly part of those applications first, parts that are relatively easy to be decomposed into simpler pieces and are relatively lightweight," Hou says. "Don’t think from day one that within how many days I want to move the whole architecture, or move everything into microservices. Don’t put that as a kind of target. You should do it in a gradual manner. And I would say for legacy applications, not every piece would be suitable for microservice architecture. No need to force it."

+ After all, as enthusiastic as Hou is about Kubernetes at Huawei, he estimates that "in the next 10 years, maybe 80 percent of the workload can be distributed, can be run on the cloud native environments. There’s still 20 percent that’s not, but it’s fine. If we can make 80 percent of our workload really be cloud native, to have agility, it’s a much better world at the end of the day." + +
+
+ +
+
+ "In the next 10 years, maybe 80 percent of the workload can be distributed, can be run on the cloud native environments. There’s still 20 percent that’s not, but it’s fine. If we can make 80 percent of our workload really be cloud native, to have agility, it’s a much better world at the end of the day." + +
+
+
+
+ In the nearer future, Hou is looking forward to new features that are being developed around Kubernetes, not least of all the ones that Huawei is contributing to. Huawei engineers have worked on the federation feature (which puts multiple Kubernetes clusters in a single framework to be managed seamlessly), scheduling, container networking and storage, and a just-announced technology called Container Ops, which is a DevOps pipeline engine. "This will put every DevOps job into a container," he explains. "And then this container mechanism is running using Kubernetes, but is also used to test Kubernetes. With that mechanism, we can make the containerized DevOps jobs be created, shared and managed much more easily than before."

+ Still, Hou sees this technology as only halfway to its full potential. First and foremost, he’d like to expand the scale it can orchestrate, which is important for supersized companies like Huawei – as well as some of its customers.

+ Hou proudly notes that two years after that first Huawei engineer became a contributor to and evangelist for Kubernetes, Huawei is now a top contributor to the community. "We’ve learned that the more you contribute to the community," he says, "the more you get back." + +
+
diff --git a/content/ko/case-studies/ibm/ibm_featured_logo.png b/content/ko/case-studies/ibm/ibm_featured_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..b819876bf790a4fe047ad85baddeba1340457630 GIT binary patch literal 8543 zcmb_?bx<5%*Cr4kXmFhnV2}`GW^fNMcz~e6ZGZ_5gIkc`ZV3Z~;F93(?hquny95mq z+?V{m_x*PNcz3sIYrCuac0YBVbDw+8>8@LSJ51%REItl34hjkizPy~2+G9KYc=%$W zJ)Tcg+F*|jm9wWXyqJ)ryTUg3@ zI>OaG-)fk8TAK=*0mWX^iMR_rI?ZS0(c+(m)^_!WA*{tJVFbpMz*TZ;n! zOO&>f3Z10ABb<&G#K&RE$pfL|7X)$gL3nsM+3C0-ocv%2FBrnd0pS#a2nca;(*0`# zKKABl_FhO$O6FgEJ>H1|Eu5VZLSV3)n;XcD8)WZj4(1dT6a+)Kz+7A$j}{zG9(K+! zcMdxzhJSmIf;*WyS|Xe+?d|CPdW4zSyEuyiAC>;s5Nr`jO8@QH&gox)dJGxZ9fkmN zf*@d9+rQ)b$J)tR4gNo6{6}ji4G#nytOj?ocX2d*%*T6%f0G|$_kZr_ui>K_At`%P z7hAZUv%Hij@DTx-S(*t!AzXqGX+dcyCm$~-r?h||AGd@wzXTUlT2KrBMLxKO|n*E2-L!=?n-25^SK?#2D|HuVDiUI#k@BfnSe?yNo^!M_=i}&&7 zzsnzP_gM3ekHu}*^OO<=g=R=zN`$Geih#NXoe-vqz zHTp1Os`xC$A8PU%VHGAiYO`_cYD_f)n{U(vN$)ZpK}eIAZ}y3$b$C?c7{Z8|;)uKZ zu(88=e84Cb&Cj`@WGv8mDODDf34xQusPBTNj#Frt>%aar< z4a@a%gLvO%9rr$*rCv1^@0bj61pJE9+zFujuAQR~q^+-fSoV3Zy>q(?*);O`cH8KC zPEMGgg+=JMY2_z$Grim6%%Y{=o0;BhzRgBJ0CdLBLa>^qeh?-=2NvSiHu^=OL^&Uy z!xAJc$TV)(d1VQmdj?)nk^XpwqI;AO7U}WJ+10V3=(O)MPn%s-x3CnwdV&fnNghW~ zEKU~TRM*pIxh;qXV|(hbLlP z{e)aHhEcp>t~OfGZ8~=}HYrk`1`aQh&Rew^MoM9|@2ZA!j(=nZs)?{0e){_yaP#U5 zs6P>JSc%S6B9B6cc^?g}z3J{?=ZQRJnKNL)rx6u39=e)fW_GYvV_AbYoyH1j&-I*o-w6f|( zS87$kK9Tm}7~M}wgAfH_ppvAJ^3+X=_O5TG<~6UtwrwCH_d+m8}7 z*k%6}e7BMKiDaB?q!6t_6_P{udr)h8b&+WCf(n$mHU2BchnH7W50%K0W{$Fy)bU)Y zF>C&2Lo-O^!*%ynZ!7;ZMzjtKr6$x)nnEzm^Fy^4cj7qu5iF!kSUjjexGt06X z#Z@;72!En+PB3N+(7)5cH>&&br~KDs2D(N=uBmyrIdddEZP3VJ141e4x)VnX_ld%FHiy$^WAORQ6-YtgoA@F?68dr zKhO`vDKG~|7`W~cUq4sWsi4&yC6+aBs4_8mZwGs488IDK_jY&o8Mbv+NrpI4q93%o zSj%Q|yVd|0#*!DoRypnQq*6CZ8aML%n=rQ9{1?rX&Md>AncW!+YDwVzcmr7{f2V zXT=}1^b1N!XC*5XZKzJqQ%GoAMawHzu~8|;&iT{R0!igw&}e&|udP(7P0<`o``$3z zEv_b{GkmyrU)}2EY5?_xkNA)cAl|VGVzoUz-1mNHk-NLmvZ(ayaYMzA6B-; z#jZ9d|@ye-U;S2Or_cM69ujs|ugKNV|9RxHP$!wDv zK_6dY1Hk&dM7^~1bAkOfBnJ9WgRSyCLq*z7+%ka^gR-c6J|;-I<3%|hPp$;XzDw!M z=Q<6fo>u$_L6-{b9%FzjC~8}1N=72KLhHZfqxpb3u4m(=ZJ&%17n%-&n)nQ-h8U+cj#|o zY&p!cD~>WWMf!AmO~!_2HAo3TGyN(WR6we~&g~Ydbk%|kOqmbU(mB;TGu^e#$eB|S z$NMxXL@EVFF&1cnfXEKEu8A@pYj#sD#h7uE%`p3tFN#bnx*_V0+6Iy?Yh%vsLkWmV zT~x1~=SnhNY1(_cfr7&k+(v!N{!Dlh+C4vY}j_ z!Deq`0%lbN4-%57@RY+mxlwh=X*n1ixyKl`HL488(qrM72DTB*uzLH-c>zM9kLE8?zjqSE`?d&Ex~(YGgD$GdFZb1 zugbRu7No61y`xh-KMQB7B+%VVJmI&AwL?bO6)4=d4TZ(!RF#LVCamG&|DjD2@!C~N z2cKHZ3(YmGsU_OI5UFbxtF{XGs+_WXeczq_T8#d*AQ5)TcEa{q+;8joBs9jy)tSnx zmMbIuy8E>D;n#Bw!f?K%<~Ige`D#siZZ<;vdRf3Atap0-PL_Dumi#3zE^lRT!0(He zA7boR{ZSc+zq7h!9AcmbRb&N9s^weg)&x*1b(`=4-vZ{JGK7@?f{8g9^`CyAZXP-Ju9J}XnmcPyVv6*u47>$AhJgc>ul=c@{4MSS^5 z0oqd~M&BT&6}2Wr=Dup^`-(pRwtRP6JcMZpD2?>h#lKq>?C_T`RXvWNrMmwlE5nmt zNx_^0+n_~mJyJ*Ne97xKJkjQJz+woQ* zeo^xtHAnWrES!AJ6hW2V+~d$c-8WCAi0)2jCKO+|$ifx-a;tT=I{io zeY?T6FQcISMCjyJD{3;oPuy@90Nh){5~443?;0&{H}rd1$(x`w>yo z#;EMBgj!(hic^E_V<*_V8Nu&Xj^BRb8_MaepCCh<&Hx45YDP+wc$IW|Ue#$yJjn8I zzig4;!z7Z`Tqu^Lo+u_!4@S1@LvX~;n#&Z*iDO~jhoJ|vAF?{$09`V-zGzPB8BTE8 z4wk*1`zCX?7Y5ELz9rV5W(M`*7j@fK?F8zjQL$f)S6$||i`=^wmyh}!;mB;f=f?y4 zl4$*$Top>KKV@#yG-{zHbagon3YLDZDC${=ZPbS&3--{JEbl2X7VL8Uj_~MnkIZMJ zFKsP!qTxvS!gnkq$J6TlPNuz2?mf#m@SfT=m1>qrLn|;SqRudaXT|PIj=Y2!#mL!5 z=+l4)r!0uDd4C0AY)PWOY+Hs<7yW5T!13zFu6ZdL-tG15pP&etJ_XSHCc1>u(lY>! z(UEv}#iwvK{=@LouC8a?uiV$5S6#_jYm?7IrySO12G0BLY2pC2+xy$j!=kPtMtqBi zKBJ6seEhptV@WzI$Yz6^T8pAEC+_p(aq01nFU)GXfU-L=cxROCreq_7w*j!}#brm< z$A=-e;g%Cua!a6Lsv^&%e)r+=WA0xGT;sdSqhV|%sR$5MF8#(YJq+PP_`0ZeCAa(1(CGEULBcBRa z_=gj~!<6&}+Vo}dH8dccYp~qp^+L-Q3sS!mqt(N!z?QV5RZQ!(M`m9*Ku4;2A~l`d zbXUOn;@1d%3RI%FjcbQOnf&KZjVvNq{$Yw{@z4isjn`n)8AJPH+eVWVG*V)O=MN!| zOtZO?JnVr3{yL&;|Lg{Lgt>F)Ne5dmMMd}^fD8vff;GXPS5~t_FmZuu2)4z|R;$>K z)N=rq(UU;E&$Z7LKJ%s1s#`f|NPL==H{`@EbW&QIZbGw)a6Fb>fM$mK9|gY}&F$`j zJy9F2C4mBzw~tgy^h*FO5Ud|9+BVyjP?FaneJ%HzvsvG*wopU%UWk}C`= zFf^%6+eKo>(uNHOyGi+4$s0AAP1ya=PvT%~vVD$kd`sMh+;k!ozvu;2faR)}uR|iC z6aiYpnp}cf$V`gfZ8haxv}`EkadzIsu;3`uhoe@rBfQ?@J~vjKA@ng~U5)&Ht6{?7 zxCYgBx(r7TRYqgjMsieB1h4M_aj$>lFK|A={x506eJa(uhOvqKqjb?z5 zJ!GG9s!2ZiUE&*xg~YdH8k!><`vz$3Cd)Jib>VI~Hgy5=@ZOARhNhf(jVTZ7A`_t( zF-jCY@AbKa8$)3Yqp@}Nm1$VmN7s}_p{>`yoqCu1ts7XT2HW*66D5FU_whWfyf`Zs!3ubF+*QdP^I)tv6-ym&F}YESqK^!+ed2qb&ER_*wz}&P zIxe_nRE-3%~(Tf+#5{zG5`j7jIUbY%0e*yEv7h_=XSwUA32A+0!&b5^5Rx zbI;CZyI{56EIL`YW#W0xA}}DV$Jkbwy|$aVkzMEQqwBkB@m6?hCsX1ZOp=kc-(&SI!WVTg``$7l@^p^!-ec@F+_uG;6=2}3f|mzr#0&_T7$>8iT#$Bl zl{ZJ(Jv|8pX^faW8uYr{8W4*%Yx@hactendn^yQ zbRvp=S$Ig4em=I^$g`#@S14~NesR>&d8RlWJA*4HQB4&7LH$yP(WZb{;VA}sNn-hw zew}`O6GS`Gb+Hs17WD~BH&d&f<*n}gD$(F8!^QCK2_sc(-U~f^X3(`523AUpj{Z<3 z7VG7E^>3x9r0>JsZF=4d@(>q_Lz_41l)N$;lHHL}jnFlOn$8ywL8A^5Cq6cbCdbAa z8N2a)i1s`jgVHOm|+u!K5jcEfCFh>U+%VR5DqK7I}J4h$Ag#B%rIZ`>Le$< zda0DMLhV#TC267Xxq2*_On3JA+n#C-=gi>u_jzV(pD1vgJcQL`O-h>%^I<0AU7wsZ zC5t<7gz7j%w&LA-aWcQN;9_CZrMP5ph&UdqbZ&l1J{(H?%HJDNsl%=6pgBVpJXKzB zl@&!3KRiEeW!U#2K)oYc&W63H^MFr+UjlmuvwA^+kRzoFq^^QFOPB*Fzaa7AFO!s5 zx*J*S&z#78%}+%?jXA@Z`xd!yL<7JeFLPKAjHh|NjTE~~n|QdjYx(N*6_l*~`Ey?z za;En{j&~#OyI9MJDG7(JBa3YcX&_pgBGr7#rDJi{5t3#VEGicbM`S7K?i>8!7YBOChl6ZM_E&h%Dx7`1O0PFV_=d=6m>8XT z)`uOiBnMMNh=-;O%|lwF`+n2)web;^89IoGf;BsF6Bb z@w*O~l=Z)|e1=~x#mLMW@q-*M9%MpJ-Snb$mxTl`Rd0VM%Y`CD;i)y5-X*9AU#)ss z`hMZr)PXpZ`ga_4u$QKx;;=BgW`(wN%JL4B?VYTDv@H2@qt-&Xe=1+9+qj1o`Kn6o zbIaFq@dou4@y_k@94dzUbw>RY?>xDbv`-Ji!mxcchq@!laOIl~#wtrSelVx8HHMCP z*}rWP^r_A|GB4eH_>(M;I;wCrG`3!<)MYGcfCkPq-@zKNe9tu%l^q8b6=?F3!)`h5 z2*4e`rx3K zb9G;fC;zh<>HDX?|EgkfF?R*Nh>2JYDyogC1t@n{(_9}5-}r8&r5Q7Q(A+D#2$w*x zn<5P=a%D~V>!L;3_98pnzC~>r^@pO3@(>P|T8e+E7SVPIq=Nfo*&vIQy)y_>gRJB$VT?vmfNX_nY76vkN0y;EqNs)3mR~5etM@GtvPlh4zBUexIm%B zRC)U5+zmfxf}t^slcrcas4}ED-`^@MoUUjvL`&KA{VVjiqzivXzySZji>c_Q+KZBT zwKs*by>x8ha-h6mLM|G^dPpAct{K-axq>Vao6@tn$g@r z<%H1#i6}U$c^>Qj*5lM%ueBmtVgB(e-9QiS-_LdbUzaHVZ>yV$g$DPnC{IvwZibm_featured_logo \ No newline at end of file diff --git a/content/ko/case-studies/ibm/index.html b/content/ko/case-studies/ibm/index.html new file mode 100644 index 0000000000..4f6927416c --- /dev/null +++ b/content/ko/case-studies/ibm/index.html @@ -0,0 +1,111 @@ +--- +title: IBM Case Study + +linkTitle: IBM +case_study_styles: true +cid: caseStudies +css: /css/style_case_studies.css +logo: ibm_featured_logo.svg +featured: true +weight: 2 +quote: > + We see CNCF as a safe haven for cloud native open source, providing stability, longevity, and expected maintenance for member projects—no matter the originating vendor or project. +--- + +
+

CASE STUDY:
Building an Image Trust Service on Kubernetes with Notary and TUF

+ +
+ +
+ Company  IBM     Location  Armonk, New York     Industry  Cloud Computing +
+ +
+
+
+
+

Challenge

+ IBM Cloud offers public, private, and hybrid cloud functionality across a diverse set of runtimes from its OpenWhisk-based function as a service (FaaS) offering, managed Kubernetes and containers, to Cloud Foundry platform as a service (PaaS). These runtimes are combined with the power of the company’s enterprise technologies, such as MQ and DB2, its modern artificial intelligence (AI) Watson, and data analytics services. Users of IBM Cloud can exploit capabilities from more than 170 different cloud native services in its catalog, including capabilities such as IBM’s Weather Company API and data services. In the later part of 2017, the IBM Cloud Container Registry team wanted to build out an image trust service. +

+

Solution

+ The work on this new service culminated with its public availability in the IBM Cloud in February 2018. The image trust service, called Portieris, is fully based on the Cloud Native Computing Foundation (CNCF) open source project Notary, according to Michael Hough, a software developer with the IBM Cloud Container Registry team. Portieris is a Kubernetes admission controller for enforcing content trust. Users can create image security policies for each Kubernetes namespace, or at the cluster level, and enforce different levels of trust for different images. Portieris is a key part of IBM’s trust story, since it makes it possible for users to consume the company’s Notary offering from within their IKS clusters. The offering is that Notary server runs in IBM’s cloud, and then Portieris runs inside the IKS cluster. This enables users to be able to have their IKS cluster verify that the image they're loading containers from contains exactly what they expect it to, and Portieris is what allows an IKS cluster to apply that verification. + +
+ +
+ + +

Impact

+ IBM's intention in offering a managed Kubernetes container service and image registry is to provide a fully secure end-to-end platform for its enterprise customers. "Image signing is one key part of that offering, and our container registry team saw Notary as the de facto way to implement that capability in the current Docker and container ecosystem," Hough says. The company had not been offering image signing before, and Notary is the tool it used to implement that capability. "We had a multi-tenant Docker Registry with private image hosting," Hough says. "The Docker Registry uses hashes to ensure that image content is correct, and data is encrypted both in flight and at rest. But it does not provide any guarantees of who pushed an image. We used Notary to enable users to sign images in their private registry namespaces if they so choose." + + +
+ +
+
+
+
+ "We see CNCF as a safe haven for cloud native open source, providing stability, longevity, and expected maintenance for member projects—no matter the originating vendor or project."

- Michael Hough, a software developer with the IBM Container Registry team +
+
+
+
+

Docker had already created the Notary project as an implementation of The Update Framework (TUF), and this implementation of TUF provided the capabilities for Docker Content Trust.

"After contribution to CNCF of both TUF and Notary, we perceived that it was becoming the de facto standard for image signing in the container ecosystem", says Michael Hough, a software developer with the IBM Cloud Container Registry team. +

+The key reason for selecting Notary was that it was already compatible with the existing authentication stack IBM’s container registry was using. So was the design of TUF, which does not require the registry team to have to enter the business of key management. Both of these were "attractive design decisions that confirmed our choice of Notary," he says. +

+The introduction of Notary to implement image signing capability in IBM Cloud encourages increased security across IBM's cloud platform, "where we expect it will include both the signing of official IBM images as well as expected use by security-conscious enterprise customers," Hough says. "When combined with security policy implementations, we expect an increased use of deployment policies in CI/CD pipelines that allow for fine-grained control of service deployment based on image signers." +The availability of image signing "is a huge benefit to security-conscious customers who require this level of image provenance and security," Hough says. "With our IBM Cloud Kubernetes as-a-service offering and the admission controller we have made available, it allows both IBM services as well as customers of the IBM public cloud to use security policies to control service deployment." + + +
+
+
+
+ "Image signing is one key part of our Kubernetes container service offering, and our container registry team saw Notary as the de facto way to implement that capability in the current Docker and container ecosystem"

- Michael Hough, a software developer with the IBM Cloud Container Registry team +
+
+
+
+ Now that the Notary-implemented service is generally available in IBM’s public cloud as a component of its existing IBM Cloud Container Registry, it is deployed as a highly available service across five IBM Cloud regions. This high-availability deployment has three instances across two zones in each of the five regions, load balanced with failover support. "We have also deployed it with end-to-end TLS support through to our back-end IBM Cloudant persistence storage service," Hough says. +

+ The IBM team has created and open sourced a Kubernetes admission controller called Portieris, which uses Notary signing information combined with customer-defined security policies to control image deployment into their cluster. "We are hoping to drive adoption of Portieris through its use of our Notary offering," Hough says. +

+ IBM has been a key player in the creation and support of open source foundations, including CNCF. Todd Moore, IBM's vice president of Open Technology, is the current CNCF governing board chair and a number of IBMers are active across many of the CNCF member projects. + + + +
+
+
+
+ "With our IBM Cloud Kubernetes as-a-service offering and the admission controller we have made available, it allows both IBM services as well as customers of the IBM public cloud to use security policies to control service deployment."

- Michael Hough, a software developer with the IBM Cloud Container Registry team +
+
+ + +
+
+ "Given that, we see CNCF as a safe haven for cloud native open source, providing stability, longevity, and expected maintenance for member projects—no matter the originating vendor or project," Hough says. Because the entire cloud native world is a fast-moving area with many competing vendors and solutions, "we see the CNCF model as an arbiter of openness and fair play across the ecosystem," he says. +

+With both TUF and Notary as part of CNCF, IBM expects there to be standardization around these capabilities beyond just de facto standards for signing and provenance. IBM has determined to not simply consume Notary, but also to contribute to the open source project where applicable. "IBMers have contributed a CouchDB backend to support our use of IBM Cloudant as the persistent store; and are working on generalization of the pkcs11 provider, allowing support of other security hardware devices beyond Yubikey," Hough says. + +
+
+
+
+ "There are new projects addressing these challenges, including within CNCF. We will definitely be following these advancements with interest. We found the Notary community to be an active and friendly community open to changes, such as our addition of a CouchDB backend for persistent storage."

- Michael Hough, a software developer with the IBM Cloud Container Registry team +
+
+
+
+The company has used other CNCF projects containerd, Envoy, Prometheus, gRPC, and CNI, and is looking into SPIFFE and SPIRE as well for potential future use. +

+What advice does Hough have for other companies that are looking to deploy Notary or a cloud native infrastructure? +

+"While this is true for many areas of cloud native infrastructure software, we found that a high-availability, multi-region deployment of Notary requires a solid implementation to handle certificate management and rotation," he says. "There are new projects addressing these challenges, including within CNCF. We will definitely be following these advancements with interest. We found the Notary community to be an active and friendly community open to changes, such as our addition of a CouchDB backend for persistent storage." + +
+ +
diff --git a/content/ko/case-studies/ing/index.html b/content/ko/case-studies/ing/index.html new file mode 100644 index 0000000000..6e2648a455 --- /dev/null +++ b/content/ko/case-studies/ing/index.html @@ -0,0 +1,99 @@ +--- +title: ING Case Study +linkTitle: ING +case_study_styles: true +cid: caseStudies +weight: 50 +featured: true +css: /css/style_case_studies.css +quote: > + The big cloud native promise to our business is the ability to go from idea to production within 48 hours. We are some years away from this, but that’s quite feasible to us. +--- + + +
+

CASE STUDY:
Driving Banking Innovation with Cloud Native +

+ +
+ +
+ Company  ING     Location  Amsterdam, Netherlands +     Industry  Finance +
+ +
+
+
+
+

Challenge

+ After undergoing an agile transformation, ING realized it needed a standardized platform to support the work their developers were doing. "Our DevOps teams got empowered to be autonomous," says Infrastructure Architect Thijs Ebbers. "It has benefits; you get all kinds of ideas. But a lot of teams are going to devise the same wheel. Teams started tinkering with Docker, Docker Swarm, Kubernetes, Mesos. Well, it’s not really useful for a company to have one hundred wheels, instead of one good wheel. +
+
+

Solution

+ Using Kubernetes for container orchestration and Docker for containerization, the ING team began building an internal public cloud for its CI/CD pipeline and green-field applications. The pipeline, which has been built on Mesos Marathon, will be migrated onto Kubernetes. The bank-account management app Yolt in the U.K. (and soon France and Italy) market already is live hosted on a Kubernetes framework. At least two greenfield projects currently on the Kubernetes framework will be going into production later this year. By the end of 2018, the company plans to have converted a number of APIs used in the banking customer experience to cloud native APIs and host these on the Kubernetes-based platform. + +
+
+ +
+
+

Impact

+ "Cloud native technologies are helping our speed, from getting an application to test to acceptance to production," says Infrastructure Architect Onno Van der Voort. "If you walk around ING now, you see all these DevOps teams, doing stand-ups, demoing. They try to get new functionality out there really fast. We held a hackathon for one of our existing components and basically converted it to cloud native within 2.5 days, though of course the tail takes more time before code is fully production ready." +
+
+
+
+
+
+ "The big cloud native promise to our business is the ability to go from idea to production within 48 hours. We are some years away from this, but that’s quite feasible to us." +

— Thijs Ebbers, Infrastructure Architect, ING +
+
+
+
+

ING has long embraced innovation in banking, launching the internet-based ING Direct in 1997.

In that same spirit, the company underwent an agile transformation a few years ago. "Our DevOps teams got empowered to be autonomous," says Infrastructure Architect Thijs Ebbers. "It has benefits; you get all kinds of ideas. But a lot of teams are going to devise the same wheel. Teams started tinkering with Docker, Docker Swarm, Kubernetes, Mesos. Well, it’s not really useful for a company to have one hundred wheels, instead of one good wheel."

+ Looking to standardize the deployment process within the company’s strict security guidelines, the team looked at several solutions and found that in the past year, "Kubernetes won the container management framework wars," says Ebbers. "We decided to standardize ING on a Kubernetes framework." Everything is run on premise due to banking regulations, he adds, but "we will be building an internal public cloud. We are trying to get on par with what public clouds are doing. That’s one of the reasons we got Kubernetes."

+ They also embraced Docker to address a major pain point in ING’s CI/CD pipeline. Before containerization, "Every development team had to order a VM, and it was quite a heavy delivery model for them," says Infrastructure Architect Onno Van der Voort. "Another use case for containerization is when the application travels through the pipeline, they fire up Docker containers to do test work against the applications and after they’ve done the work, the containers get killed again." + +
+
+
+
+ "We decided to standardize ING on a Kubernetes framework." Everything is run on premise due to banking regulations, he adds, but "we will be building an internal public cloud. We are trying to get on par with what public clouds are doing. That’s one of the reasons we got Kubernetes." +

— Thijs Ebbers, Infrastructure Architect, ING +
+
+
+
+ Because of industry regulations, applications are only allowed to go through the pipeline, where compliance is enforced, rather than be deployed directly into a container. "We have to run the complete platform of services we need, many routing from different places," says Van der Voort. "We need this Kubernetes framework for deploying the containers, with all those components, monitoring, logging. It’s complex." For that reason, ING has chosen to start on the OpenShift Origin Kubernetes distribution.

+ Already, "cloud native technologies are helping our speed, from getting an application to test to acceptance to production," says Van der Voort. "If you walk around ING now, you see all these DevOps teams, doing stand-ups, demoing. They try to get new functionality out there really fast. We held a hackathon for one of our existing components and basically converted it to cloud native within 2.5 days, though of course the tail takes more time before code is fully production ready."

+ The pipeline, which has been built on Mesos Marathon, will be migrated onto Kubernetes. Some legacy applications are also being rewritten as cloud native in order to run on the framework. At least two smaller greenfield projects built on Kubernetes will go into production this year. By the end of 2018, the company plans to have converted a number of APIs used in the banking customer experience to cloud native APIs and host these on the Kubernetes-based platform. + +
+
+
+
+ "We have to run the complete platform of services we need, many routing from different places. We need this Kubernetes framework for deploying the containers, with all those components, monitoring, logging. It’s complex."

— Onno Van der Voort, Infrastructure Architect, ING +
+
+ + +
+ The team, however, doesn’t see the bank’s back-end systems going onto the Kubernetes platform. "Our philosophy is it only makes sense to move things to cloud if they are cloud native," says Van der Voort. "If you have traditional architecture, build traditional patterns, it doesn’t hold any value to go to the cloud." Adds Cloud Platform Architect Alfonso Fernandez-Barandiaran: "ING has a strategy about where we will go, in order to improve our agility. So it’s not about how cool this technology is, it’s about finding the right technology and the right approach."

+ The Kubernetes framework will be hosting some greenfield projects that are high priority for ING: applications the company is developing in response to PSD2, the European Commission directive requiring more innovative online and mobile payments that went into effect at the beginning of 2018. For example, a bank-account management app called Yolt, serving the U.K. market (and soon France and Italy), was built on a Kubernetes platform and has gone into production. ING is also developing blockchain-enabled applications that will live on the Kubernetes platform. "We’ve been contacted by a lot of development teams that have ideas with what they want to do with containers," says Ebbers. + +
+
+
+
+Even with the particular requirements that come in banking, ING has managed to take a lead in technology and innovation. "Every time we have constraints, we look for maybe a better way that we can use this technology."

— Alfonso Fernandez-Barandiaran, Cloud Platform Architect, ING
+
+ +
+ Even with the particular requirements that come in banking, ING has managed to take a lead in technology and innovation. "Every time we have constraints, we look for maybe a better way that we can use this technology," says Fernandez-Barandiaran.

+ The results, after all, are worth the effort. "The big cloud native promise to our business is the ability to go from idea to production within 48 hours," says Ebbers. "That would require all these projects to be mature. We are some years away from this, but that’s quite feasible to us." + +
+ +
diff --git a/content/ko/case-studies/ing/ing_featured_logo.png b/content/ko/case-studies/ing/ing_featured_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..f6d4489715aa4bc1ecbed9ec6b21531e07a02ffe GIT binary patch literal 10305 zcmcI~bx<5Z7bgi0L4sRgg9lmM-QC@Faad$=3+_$`PH-m>T!Op1yF0-hj(m1^Ro~rT zcQZB9J>Bp3tlz7eu6|*P@)9UW1V~U&P$*K8qRMaW{M+M;@c!-nUMKDNt-*(gX+l)M zW)N2+M-Y^dDcBeUkg_#02PuP$Ox+y@K)g^;(2ka>nh;GnS#A@sEzsyM4baWj{tXQU z#Vg=uZ)9Q(f&h#`=9YGRq-V`-qyS4(K2i-fIYv2q5s-zYq=zF&#Y0}z#KYQz%al}r zAHeIz{l>r+1Tg}*+1l7Sal7%6{(~>~TmSDc11aDiBoJ#p(tj$YDW?b!0Xu>KY(RE; z6DC$h00$S4iJg&^jfoDx%*e#Sz{tkH$WG75#LdXb&CCS&*G2j!&C%41TUk{6U$Wl5 z_(&}v5PNP023J>CpeqXy>}bxw#KpzMz{t$N%uN4ALGR>l2QhM^w{s%PiVgWr7je;4$Z@J$IQ*Krh6V5mH8PPMcvoq7Pa+$Ky8!>V+zd4YR8N_UC%w!DW`Zu5dhG$}8 zV-gi*XXjw&%FYR5WM=~Uw`|Kd+5f__{a>-% zB90&<2-r~-47T}q6ewDNAz&v9usuLTXU;Otk ztl0k%%kZWQ!{0#pUjpafLvPXexBH(oc>DNIB7yAQvd8f)P0GmFWT2qP-#jOz>b7{C z0q=&TwiKL|vJk&=wJL(}+chs#^_`F)IVpOvKR{9`TrvbO$pdo$kW)MW2$4gC6{xdA zSS868YF0*)QeM-e3?Ck}F9c1B*0pDRl$AyLY z^>+)J8XE(cnb1>`@j_)kh*DC*k^aA*`pfZ6R=1$n2!-g$Ms+-Af^2C!zMqK2|^qi#bjR-%5`UM@?aY}Rp^;DV8)5FJ$TqFBymJmA38@Y|v zrCxlDhH24aZ5EXk;-6E+Rc$nX7ME%Cf?KjUHs3KwzB^s+V!ea9S#7vQuwy>&>Uj$8 zP=uBeq|Dp>X1K@TuFB|~HY8~JaCDj=#XzziCdvng*0CgF?}O%(9tuMyjV-F_xJ{4^ z^WiJb{y2PHH0Kk&{C@s~cNCvGBYMh9&JV_yGW1C-!e~Gx4yi)+L36)!?>NA8e;t(? zDq}|iO?ENbgJk`M4Mwf!Mk$$f4Dyt@zuhNjO4CTbppbBPd~|0@63o5T66|I^e20%J zk2eK(L6-+jOfKXgUR}4}3=2KeP%N_6x?hbGAT0%C@BK2dtzT^#RBSp+$`WV5ONc!C z`5IN;VqGd3+ITGcS=43%z=EiOnrjU0rje&35FM0(du)7u!p(XZHR}Dbv3?%Qaq+`T zFT?T-w4^uPO&VE~hUre)s}cI86Sd@;g1-d1u=x*Xa~r7~ER2;lsFV*QkBUePwuNU6 zNxdzPMFzM7rV6Oy&=?=QM#)M8`~qbzCa`x5?vO+F{>bZliRV;0^JXQ`EU}%}BYkul zd9{uQ?~ml0k?x`sxb_w5_`LRPaLx0+OCUv`MsK2%t%#JSAHU^!C_N4d<&FFSMRy!B z4{z^pIqN*}y9}FhFqW1O=c|PKJ{tniuL0N3TOhaQ!#QQ~{ZjAP4O7Hj8wLLc25uF~ ziATo@igms56d>X?3VN7&3Y}K@OOPmy#ISs4bHcv0H!6|WcqNzFwcsXnV!_fNoa#s7 z^e+#4;Yd7FEouF5?B%gJw=yZkf1LbXMQl8}R~iALC&qx67(cf&tk9{{PyWM@)HrI} z&rXdPq2nuUC5URt*!CC-UV9%0 zRTeV<7w36ugn#C)hUWsCuK|;11L+O}x)Hq&fS6!o|J8DpZQ}Rx8jA4=j=?VPXfQho zQp(_p)H;`=YgU^TFPJs;LWS9gxvOV&W#tPK_pug0v0!!lc;1cr#Acajx{f`bI%nLd z-&Q@mfGosJoo3HuCS@AT$ItiWlRI|<=RDVEsOphLP=BBYoAPY&JL$hz+u*-~HC z6CkaTW8%x@<2-~$WnxF!ZWst8k>ozLG8Z@Ts z&s;A>e_HZx$^%G~x;{v>%)sci3R%#4WV>;BudPBG5cke}YH)uzNGh7d;Y)h;$@VT$ zgn=Gvx9QB~)IN2Acjb*dVpWOPxn$*b>*^svWhPJFcgI9p2~}eUEK(n7%sh?AJI%QG z(uP)3`l@obXUsQVr^UQ9FzKG|3|_XVB9MB_UBjuA6x^;yuQaz54~(*G6TY~mqsWBp zI|pN$8z(W$7w@$ z6c2W$C90W}f3io#ePYYPfuEo*{zMMX5K6i6@YzRSBiq;fCD&CBdR9jI(HCI7di8uB z<2h1w?nJAbNva{4m3%R^dT^pM2OJc_ATd)dViAq+Pg%rauSjpVo~r&aLq%?PZtqli z(`@;x24wwVayM zw^m=3|v5ezV8=+{mR?Y3B zS&HP6#vhvt7Er;?*bD(}?5QYwJ)9_gV(eWRTcd#UBMwjqCk6Gi`|fqkAE(`uGYTN z2UIafii63WlB91Lo0aLI0U;M1qRpfPJ)%#oWu?#wmS^R4jEBQ>xoSr-wRPIeY z*bd+0FlQN(8^sS)w95;|YHpO8mW47h8yX>BVsd=#?tiUnjM3TLcIY(Sk#~{~$E%xD z5AfmmJiW(HvI}+ZU*(Y)CEZL)UoAZqSh8920+5I!^!ON+z27bZs zwdpaxMAuAW4`%a0`&Kd`!sQ-@qd$Q53*IMwB-GueWee_ z!7$%osLZPS+rm>+jiLCNJQeg>OLwyr&N1_Jwcs_{)rKuPn0~;;$r)d$?Mc)WGI6~B zgWwXRfw8-{cgU6L*$aew08!>Am9rP^yTnyDh+V8jlza~KQuT*+krsS7e+4eks_AimkW^Hwmq^((}DO3W>{F$?CTfbA&yrC<$e%M>UJ8U0qV*8Z>KHo84=kF zs1)6sX%!jDgr-DSX&d3KBoknI@|c6}$MvMl*(!rZP=|>yoFGv4+wwlTsE__Zjg&XNkc2%pxodkdAP~nzAcrWpc(1QHlLFtGEx-`ktQQ zBjepl^pZJXn96Qc9AywovFLd*E*VeL?z)Bs56`(&uFsvY6tzYY@OiFxRn4Qb zR4ZYMDKCFe{QP_Rl!A4zm_ZD~z(;bS(G1 z#%*4bp6?~+Kl!VbRpW>eg{jbi`+cL9{Z)?2%XSh9L9NgQJ+B88k3BA`Lm}z}!lXr2 zu4BZM!gTJL8zeKjVQ8Kg4~=*t=8r#PRd$0omBY_JIS?ON=S&{#(d!v0$C;zu)K|mi zBnG_sCKVYyVO~C`)7vOX>GCU-bpFpaPgms>_YKRu&r#M6Ex9L7sI|JWw{82W{c6fn zw{Zw93*k7(g+Tx&q^{P^Xa~t~v3%*GLs=f{$eN*ummwXW;4-02kj*Pn?P{=YcPE3r zJ$*iX02F<;i-f9efd50mr<1@R&*)~_#(4)^jVM4~n0MLEOg~Z5b*Y;XljvCh#|+9T ziX|kGZcoKWz9@oEA}9V!&E8($_0JABO*)W+!mZ=mosdXg8?$MX%6K)8x3YLWF}WJ0 z^EXp;b&4IGt6nwq;H%VeAKEtHl^RhoM7{Je&*MVdBA^Afy5jI$Vo^(L4x&9W6f58g z^xh%j%CHH3=h9@xY^D+dD)q~fdlCc%nU5cP)8Y9WwB-l*!1_5*D&})>qY0Jt=p(Tb z^$j@JG~3*vwy=Ck;4q$K?>`Bh+T$46fuw5q?pd8pQS%)f`o8OaZ>zuR>$FJ`R}z4;3B5?!yH=E@67y)+A5g zY;wE0U+_AXQ>c9#lPY(F_7lEnCUZ?J@J-*T4Q7F&raM-M? zlZoX)Vo7kY=qv(^qXka_1sRCe!UqaJdy+sScHd0TVvy*`hs|hX z6akFUbW2Fx*l!fNQU?+zObMP+WZG)FPb9Z&fX^MieQdar*|B0-di%#a72bbhX<5_d zG*uTD{{Z#gOz zN=6MQkD?TOYqUl5C%5D?2u>F`Y^O>#z#OIekyE&7A97&GSNfj9_W9tR=QbuqS_u8e zmlVGj2pN4-M(4D^!t=zzbnO@>qd?|buwX-q(AezIi+n{$aj0o*YF}=C-*((_5T$uT zd#Wd~?IQaCyK1F7kd2S#H>2NP>wUM2;6{NCggq8>w19SR*g+tnHvxSm) z?pJZSbcGXF`ApXTuee6RRt4K3}EMI-L}jqLaAqt8K10na2>S8a|~2mrnJHI zWp{vNpnZdT%^A7k#oXTr;_2Hcy&L&~1SzHhay)}n<5(%+DNl6Z{6+Y7cIQdv(%5z z%tZF?QnHBSTN&|APNY@QHm|gnH>!)!5rPC((;|DA{+#!b4Ji^!d7{~UjUlvpn206} z;=ti;*P-91jxaqK0qS99dsaRXG{yYTXSUP`(M|4nT0T&_gG{k#rhVX;y}+1As$%u37uQMrKaQ+3y|>@>n3D5``Jh@oe? z@%_GN9VG~p(pi*HV0+J4i}y(;x1HR9*{cT0WK4uY!Wcu|OUB7)Rk;AD0&6SKbM4gN zgb%*y)(GkNLLG6B6G}EZnc@fj?H8f5p@x56kKUS$3Qa>VKK&6;H&512za2b(9e~jZ=FCv>sizG9LCX1L+x!r-e*TJsOh>7fOtlMh;hL8r8S{=LeCdGNU7ic$-xZ68zSUH`(uh2mvB_ zTnU?b60+MI5o+PiKR5P8L*u6*>y3$2JmW7JWgp1b`bHTn`IdVeI~u5rC>TU?;C^>8 z-s1B1=NieU#jG3n3Cr@6^6bE}DPXqQk=H`KqnT(PJVGQjuiYuqVA(HxM(!RCxp()7 zu7A*_Nxb%{7NHeUo}60DBMv*;T%k&vu}ajEdk7N-v=?lSeO3sSb0vifa$QFywMd@)sp`GRUGXNzf zQ+(7)R9e3!4R^`3S4B0r6T_Mu2P(0|43rv;O6;j?E`854yb3^AKI!dfBUp(W|5@{R z>9}5(Juca88vd!!uX;23>C#zL+v@XZ957KO9oT+TZgslcR_XLOcDeGLNRw~6lb7P1 zCeFi^IStw^B0533QClpn#kYi(DA#Vb7+)JP7^EL9+N(#~i5Ba`R$h?S6(-PCZNQB3 zJ5s9UTDjp2HO~9REIz?hjj0&}{C06IPsn;fmSSX~4^Gr3!viE8W!~ z9dmTg4Aj#Dr*$oSjT|LLgJp}93YuynQ4{=UC!Dx4q56F4IXov%!#NMul8~=(kek+` z{1$@Gu+#ULgNA>PKN&Q2&FY-tr_<|w+3gA`Q z*g|8rHCnuyy}8M{C9(->?{yzK$$mT&7*OLre~mqfS*(RYck|O2dzddeND0sPT_*NE z6lCSpEtkerp0f>;_HjJdKtK>a`g@*==gl2|`HhSMGuo)JG#ie!b~C@(S?@Y_G6zxz zksD#nzU&_QViTz;)343AME%OVggh6Y%+6$cBWZX7LUIwJ=iF{&U(8TcRGQI0$9a1C zj49=p7G?=tEVMD=Zt0EWLeKCS1aw+l1F@-w%*kaLc9WIzDIjaJWaU#rNbvhtAVT{3 z@uhk|N0oY#V?8^qmcf|K3zh{T$=4o1lbr|H|)M1fGKISEOl+ zkbIVGcUkLEa@e{Alj+Z#FgZ4Che&86!fhRzYcV(LHd+X^@8`%WZ;(~p;ffvchkG}Rf<+WTAtrE;6BiaYcl#Qgb}o#XH|K_*qR1{;++Nc z7N_;7<&wk#m5@Gt2~xHq4=okFPX{y~_d^w6# z%SWj*_2Pqt>@6IpR=@Y)4DXXyC1k>&?$eKda^8L|4iTvOT-WgVTqa!? zxSBMGHHMLKykHjEF*ujveifO*KGfFo+WcHwjZ`WqUW*ej$%=6n;ZiXHI4u0YRXg+1wb6#}c{RRf!-5-z zo<~n7)wriv&z!WgpMv#izazix#&4Hm2qP)C<&(QD}G_vpvJ{&F9fkqzD>Qsv` ziXs#yLDphQsyvG{&tAC0i^s&%=0fu?4S}pvm)1xCfi*Xp3-zpyp;T6)k4|LzLXNP} zQ+tH9=V;k1k!kGRV|~}ytviQ4oSw?-t)~)m7!H5B8*T8eLN$Y%V=NZjj}Q(+-DUjg zc&i#=M12LCo^QJsAwScvy-vE#V|gdG@PDn2p{tF za$c8pBla{qs5k8X-FD((#83M2JAx+SSNuz%$*b(G-&MT}EG&o;AYp0TYqmXXXk6ss zS)3GG&Qz1HINivN{2;gLN2Xymwg|kuB9Eic69$qMDnD-w?V?a)?(Z1>babuVyWixc zx-`F{=ssTZ3Du=issS2bUc0nHJYGhB#~H|KW_fTRO6RZJ6ase&M`ofvRY8v3mcrGb zt>+LNU4B*9s~lXQd?`+4iL2kT>rL1;*f^5keAo0fxuc(_Xcve-`_o*eR{GxRR+#sq zK&d}8-(e|A1cKdo@k8xMB}7RAZnJhwH~aLivBT+o_lgc9p<*s3j&0#l*X&VrNVeP4 z*WETD6r7-(RX!6S^?kO7?@cpAWGkmPfQy(gByvsJB zkz}EPiV=gxpwm>nK)uxE=#)GOry0|^_I%Tnc$xne!h2Q%5BrXHjvI&hpq(KsU*ahV zXq=Pz2D8KT@v1IvY!lbtt8Sd69$QHUw}A%2X25&?nqJY?><-bhgeZS~B;!w)2ZtF~ z%IG@s^2_Qozucov_0k{~Dt!YJ;j5VUy{o!cu+-JX6fAKj7yU!oEY4W$HFMu!%(qh# zU)F~)BKHLl&my}Je5c*vule&z&FcwcKIN|KAu{bL)gjp?jP~DMcF2z!4{O)-eYM3q zj*j_GX2qc@Pdx{dy|S;eVJNdi)I)`JfizGUJ^i4vI1-y79a*)lo|-o{OW98Wdp`PX zDnq1Mxw(jJ^2dKXGMV)4q1S*>)W zN_QA7Hf`d~4TNR@)+0IH{YYN;2U9nyr-`8Mna}|+(+fkZJD1EzaTlk5;IL*_j2$=F&W5QQXyg#vxzFhQ$LZF6@ zX5wA^_n;vRJzc@k>ck*3;&kH>Tv|38=3 z{O@HvZ^GWnc>dP$yp{3%r|QP^Z-vj>ls@ir78Ep;xq&jTo}H22-=aP#F?rDnVMG7_ E0oEjKyZ`_I literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/jd/index.html b/content/ko/case-studies/jd/index.html new file mode 100644 index 0000000000..ee61da9d14 --- /dev/null +++ b/content/ko/case-studies/jd/index.html @@ -0,0 +1,4 @@ +--- +title: JD.COM +content_url: https://kubernetes.io/blog/2017/02/inside-jd-com-shift-to-kubernetes-from-openstack +--- \ No newline at end of file diff --git a/content/ko/case-studies/jd/jd_logo.png b/content/ko/case-studies/jd/jd_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..58ef32a3221e46f261792ca0f6d093ae97b46c2b GIT binary patch literal 10738 zcmd6Nbx<8ow;=8$xCXdbaCdjPxCILi7q~cFT!T9V5AGfa4#Az^PH+ht+%L}Z``-KB z?jNsqtF~&ZcB*?$ch9jteR}3}_e83x$YG+9puxbvU;^c(HQwOww;X^1_jbQh>o$D@ z#I7>Bu9}XPt{$c?AQ%Y?M>7xrXm4r-(g2xScsUP)gkfOdEx}s4uDZ%fg659)tfv3q zuzK1%y-~x!2#a|-nVQ>zTmfbvE3ktI)md8?6##4@LZ!o_%&zPt1+oUqd%J)%y;Zc# zy=~0}EU3gp0m7bwZw%}~uBHG_dpie+pr;7czxWEimH%mGqXPU3#no1X>OYjyRaOH? zIl6!VJgi(S=Ik6C0A5~J4sIS^UJhmeCp!lx8~a=0W#Ql!B;latt@`k|z@p5oA^<;5?Q2(2Q zGzenu0(No*J30XV;b>~+=;kUy^`_~6biv+9S^2*hJ3#)^P;Vw<^E7p0<6vcHv$y}J zU;n~}xN3m@ryBn)Hbl$I3B;xWf;hUln7@sOCH23F-)#547xWL}n>K=~F5tI8F}0I+ zGl;Y#y;{P`-&;jCV>R=A~w_Na>-2bGN z{@>CHO1XecT^(Jt93Acc?Ez}mj;@XnYey%5lqN5LUI*-8;phQj_-ANXyy_PON&r@rMr=ex6rTFR`as?H zw#w=UJ@NP9KKqn3ZRruf+GV0C{PLxQ#PD z62}OWS2b;uM<87Sl4Je(0b!Uv_67vc?F(mC>lwZOE#q&Dzhj(4PgvqU) z{`6ny$7)w;n+h6PGb4DDi5hs$vWNwL?sPk98i^VafLO*8aTHWkET2g1)fE@>5MjVj z2Pz6?zvHL7`ay`_ih}$ZE>k85t|{=_u&pVfC*nugToc6y62#=;M=-;gaH#l97cf--dSgy0`h6y+|YW<&;%@s+dO#b{^H) z;LpxkT}?nmMZLL+wM^2hQpQ5WUWEDMmS~?XOW^q>ru-U--&?O|7x_JT`(efXIr*>U z(MN=jULxd+M9E1hylRXozo-=J6x-Aa66QnD1AcN!Aa4Q+B-NI=9EU;iNBUKG2g#v+cpGEozB_Zj^V53Q9m_VqWI* z&Q92c91Mvaq87yl!8GhdMdVjGG`VSPQL1ye$Zt!bNf#WKuWG+AU>opSb7pNCzrpU;5=cg5?72sbL>6;))+$}_HMD4vL1y4erR+{DkRO2PxsyHwoo4- z1DRBi1zZZ4priCF?b(zfF&>~e6L2RTO_tlf6Nj$ zVt0o@Q@g9-;c8MWe4Cq>TLYoo0 zJx|`*861?2)wX3Pk?H{;ux=0T4oC+wgMkI%xJYB$f|8c~(h=XJBFft(f%Oq#5$1`- zvcl%r(C)q7mGvIPuS6s>=?-yp+SLaFst&zV^pXZ2_Z#k?O6w~%b`K%$SXa$ z6r#QQVN`*-PHdQmBuG-oSVN#UQO9kqc>K`@m8U^{$Z$csOy^Lb=Vi<-w z```OFbc>)t>a@Qk`B&RguD(zY$kC~0ESlLOv-p~?ks<)4uuSTgbhWX`$S}3>iQpuh zWn>2OrI35lqa$NuBjb<=4l$6D7#_SVSY**p+JPCDc_ai2ASo3 z;EzbWTEt$8NIXFrMX#eDpl5bcDGU{})}##=x=c0fq#+9M6yC&n^j)urQZU*qRmv9h z-T8$MlwUT|(}6C|iVw~Ow3~D~Z@srIx`<2}|M8?4gOdL0Kisl80?|kBDTb$P_dNc# zmr79D2eJhN6%;DUt2%7f88+DT0Jc+fEsQ zE(%(OM)@vES_yGY2V*fO3kyC5lU<>@t@tiXw_>DQ5T06+#s(@v3mSWi4eB|$=LY`1 zLN+HHEjIQv_5^`It^qxkwVgFRqi_*l2BC3)pNqof-~R54Y47Q6#5CMSr1vuo3>DaY zZH|8)E}5!3Bh6Ghtc&Ea@v>s3`}^g%w7JZ=)dev*@MMj5T1vz^=%E1YQkt}G1AbmR z0z~r8A161>lFG{GJE$SzS*uzhNw}dXzC1P5B7Q_1945#c=+x@966^?i$Ah-tZ$^pz(dB#9mP#AzTA;RF-J0CUz-lxNI@c~qn zJ)hdZAzt8gBtZ_najNba3-< z>Wx}(H*=U4=v(7^yjYu_o-Pgu?}OD|acG902ZSM}b`Jt#_0rRm=v0xsgRqgMR_Epr zY6n>}(f31Vw| zx`ge;mY^R=rx9GE>5GtO@hxY29d2(34`fO==>DXwiT0`V7hp?{eX`V=`fiivb($c_wV2G^72C2SV>xRQ3{0mb2Lt* z)u_lwSd~g5*;BKyP=L>-bP#>-4LX84jhUmPWBpRH#yOJ}a$9S>DeMz!oooaECBcM! z6M9cddKjOU(&Jn@nU#>9EmB5@11zVXCGI#mP7wgVSeX9BqtTL;z-|y9Sx|5Ygp18H z!Tt^aVoF6v)ScQ39h36eLgZ*f9!dRrV$Ld~+p9XR4~vUb6w#^9-op%l257zI&7tvh zV0rMT^M;6Z@!P@o;G~PW!OzD2WC9yTT{nY>4^O_4Es{5EIVd^{L8?hlX(@IiHMu#V zPGLgs$4~AO z41<`cC^-1+Z{JXUh!?8l#vew}nNYSQipvQ|lxf>CFIWqR^%b6C!O>K6Z4iZtSTp#+ zql-@?M&Qc8^$+de@E^xD?d^@Cp|Pfm#y;m?auKoVT_buGNfszcwCDoJHv4+{V4>JS z^y&qO-qyOHvewS0R!|Azm8S8N@v_AOIr63FQ#cwhJ8psgCh21WhFe9-@pe8H0}51l z<76w?bb24#w^LJ(D3Azp^0`Q3xW?=_80uQ{E&i_Q>o^Yx`((Wf|V*e|O&Fb+spf;?hbMXatD5xNx?N2A$|%clUIjoScZuRnBL8{R(3r zbcJjOzwD-M&*9wx%bM$oEh9ifM~i)UV3}{7yleU?M4tel(G*5fMXrLkvoyVK*}Mcj zU4qQ>{8q~3$hMr&f-S~EAt_O9T(86HeSC+HgF3N|udg56coR5cQ(gDbqM6!lu1vij z{>cJ7XyNAgRKfjP_Z{<$sh$(bJ|3)593+bu30$K1UYtB4VFT_z65!mbQ`4+3oBQfY z@eVkP&L-yz%h^ks%EB8np2Wr=!`YxcVB_kmCc(G_>7{~C-#aHkuvTAiIlrv&lYJ6rQO8eJB#tXb}Vi&r|AUZ&+Jh%sc(pSlYg6~E{%Mqwe& zpcF*l#it&9m?CXqZ?~%h)-mBxwx+lQBCpM?Vi;}K&J@;LO&Ya$Cg>LmT%~JkB*QVE z7yQbBr<=mGBL1Y``jJJ%atUX$Cpb3XxpX@z7Eciujg|_Do_i?8u#q3y#?s(rGRu4k zM}Zcsw@*SE&(axdrVC3xt%8gns;QwvXMoO6>zcpAgGU=Cg`F9j>@G*CpvwLF;z%_l zXC0?rX_IRtBMyLrw>5%N@H4w4sEkt6QWp&YH}6_0b(D%nC*NnCzD z0d!2^q{^p(2^5U{b! z2*9@6!LTT?J=a_d^bzBTu%G0(7nw z2yGMcRZT+|$wHDM^suiqt_(1v#WG*4u=?J-I^_)aj*x-&9b#d=YN{sEb6KCzlu&bc zP$_W}$3a`ucb_{x`!jz4LhzDf_4H)M!cqymG$HZFq(GvX!?9euYuvkNTskEpKKJAA z>44acXOCy32t`00(&bMB)^hP4tDs3)>I&hnZF~K3>gvC+cY4Q_m3NEflnJitG95E` z%?E!PWWy97nwe$rh}=-Tes)_4PQ)+$`t|QMrfL9uABYQde9jKWN2I+ zy1G2gKS0IA2JkysuG_uwI6m{k7%ik|pY@)_d~VS58tpKER1!t@i?#MP92~~+5|b25 zQZ^i>!hr9tY|fo~C{?`oQ2W}0&;vus!mYl2|2{T0hH<{%`RFiN`Jdy*a33bY&A|)Z8UY< zek|lFc;Xy?h>p91lFcZQh<%gqdzq%oH%~E``teI%LDWsRkPqcEtSdGk-@-A0}W*_;HMsjIB5)!uD zVtHa&aB|Lb_<-6$gS}u@f?9<3)B{7(T%n6M{|7Q|;iZzB2gKyW{Oe^kz1?z#=c}Ah zLv|tzF3T~`odoPfQ{spZh1z%5Anr!AWJ_X!u2gyX^ zS#==&9g<2(u*~o%K7e|x!W8#j3?p%15jhPxff^6qqjs8H_jx3(Nyf%q=1){y$A#+K zU#~S^1m+z$haZuu{4VLbqi2Xn5C@#5y~of^7MJ^6c-8_$aefJwXd-)sF4d?OmwWr&ou;W2Wl%L{p+a zzKB#&{sye1gluUg3Ir(^_^B8*pXu!a&Zs%Y=&GJ5=7?u8+b7u+rb*4GpfN$1sFT91 zAp=d#VmObC3Y#M2Lq56Zl>UpxIdtAe8WJNi5%m6t30@%F?QM!qolWW{zI9rp>1a>? zdiD<(C+js}U&%kNjjZ*pmKEa(>#O|jovcR{Op#Z_E{9>Uk?K}OZZsXHu;WP4ijN!9 zH}ebP;99R9gV{!%8-09wRvB9M_A+9zrr(UCbuwW=;J2+W-xZJS<&y1ZRsJ7br!Omw zNp!bp!IHc9;+P9I8X&71m4}xnA5(%{s2E6jwef zgUNgnew%Gkc*`p((MUp`93bRm(6>EFUoTtI`5+0EHY{3O zZ?epOPPz#hC4B8i!>Z79U`y#dr3L+|S>6y{@1(Mk+J`c2Z#6|ncKIZ79u6Ck^N#M#C#%_D+SI?w2WXux?kPyRFtTD1jLdWNV0o0) zM9}mz+~X!I6px=e@n?GnOB8C{3$qq0c$tnJP!b}jVFaVy2}R}iFke0EHqU+%{xX4? z`m8{^_DuW{xNIXjUON`-wl`;E77@nDxsQYxxg0iFcZpUxkjUqqtY3wJ0B7Iio@yF{ zBt?U#QG-^jAY%adiZ+8lK2P)2#U4o_`+N3F7s?)8DWIR~bLR_z@1JArC0tf=__1xo z>4j6QJCux~==}CAHE6B-eusMM(X6k3LE(XHk#qvweoP5FdH@L;Iln=J$5~EE*kH`} z{7ENjUz`l-R|jq^{n5RSUqAwKigXjbxQ=%A*_8<-zZd&!D_K2 z)%i^5+-BXOo}Trqe<-JEpSl;Kth_Zyz%iz-k7=j$nZ@seB9?C(6xBeN4-6-OQcvBo z+UXbG-8Qd6^tgRw`;Duosz+&LP=?B}>+xk>#AoI6=Q;f~+S$@0bso;1i=d7Ukpn33 zYREZ@dnlIO6?H~p;GNYifSwuk(|iOTb@9m5lpo0VDetrx(9*d&bJ?qSn(c!qO7D}x zJ#JIFH^kPp!v7jYZV)4VOz~6-8n=CUmVu*4q=1nvpC=qsc$$aL$hr7IgqAj`*DVtS zj2Fu|zb=H&kst4iZ|s7K+H=R-GgJFFq??6h1-ZAa*)-mErH`KI{dAds+n;*!=XREK zv9Y~Ueo8|Ctp+-NN9~XB-r{7FfX)Ly|JoC0f!a@uGoS{#jsGy|u<&#Qt zX>GY)l{S2)Juj*9v@cCoPC~UXlK_bZ-oz87TEsDvC_zPKDN(XlJ}RdYN=AD$Ql1ID z0mrL1^c24J3AH@nvBTTdA?NKYqMbP))CDd@SnqCV223 zlHy#OtW}NATxn+r)xN4c(l_#rwe=|JcxZOGWQesx4at1HGn=8vJ(2^HqAq zT)WW81J-NXGrn5nb8vUb1%hbp99}2AYq?mMowA%CiW{b< zCXKdmdB)i~N`8&V8$)rtGVly_UY6P*&9>V*havsP0k4KhDZ`4}MXjN7C}3to^g zkyo0k1@7-p)Mu`4-so_MdqP6!BhT5HkYq$IFDIE_(@1JIN<3Sz2o|3A>ei!rPiNf{J6o z=|1LV+YGuumaGB##s%5jMhL1NUm>dGIXwJX7A~6+b-m+DX|eCAzmu(zdQ=UNU+CB+ z1(St_aN=pjlVH28Xtdw~l1g)c&#%I&W!!&{v`P03S)EK1gz5B9)XKF8qGhTy0)5iN zOnVat&^2dg8uf1$YYX9fHm7ImQuk7w&4&((&G_@aZya%D#yB<3XlPzu(G3m^Omn|& z9xbk~FRZR+Wo7uf&v>5LZj|8LENz5w=~!BEa;QtmNpdq-m;Jo&H+&CqpkvRX@szZn zGBRF`dMx*U?CJiDnf9T0Ty})pDS+(kHlp(jm1ruE;;yw^c6Wxvu%#~XAl1NpR>M4? zbCn$ucGVFdOe=CYzH00UJzm9b+z}k8p#g7ReJ#eH(CQ?Y#*WpWvEC$En4h1>;pzM9 zikFtWFweKMgVC%jD9Ci;?xW-yGo1%*F+@W=HalMK$jgHncXihenS!qQKR)Ja*vQC< z>9gB?;|l)JlG=3U1IVg_^+k772QD@8A;f(!pZA_Km7dXJFCtIJcRPxOzek&y`WW_M zu2{S)pB4*^J75y#6qdxh1cr}aO-Hq5)mPbG2mL(pdS#>gl_bY#e!h)GmMT`Yii+sd zQ&LxQkN}=G=KPIVeTS@*G9zPUb-S60xzu(-L1MO$yXP9Kmjj7c@Y~rz%yo?Xx6x73 zHKrC7RmG(JbFo>xYYNtimR)lFLIOdnZJZX9xsy?UlOXkEy6fb6(Ssdoa>vy=3|;wQ`YwE6Y|_U@?N^b%BQwCX9KW(c0`qDc6kcBkbbN&8`G>F}HuHX*?=_vRvd`-44 zwugFZZEbAa+YGGfv&VTshZf5F*JMtWPp|<7zN-<~OQn{?JS034*?MS~?sD4|Vri3Y zfi8~-+_y%YLoa#YVYiX%hCs)k!YjHzY$N<5R5PsPB0NV%NMBnY4!+;d$3Mq!-rx%2 zN2wL}8}|QI3fPsI|3zN!it|C|@oFeqLvCL5vVWMwFN1fo!{Gex34Y$Z(B4hoZ~dE- zW~I;m>5G)x7xOEE1avT6Od5XkK7qwdeS8np+V{cBHMd268xla3Lc%S7h_Q)rOWoYi z;A9Yj7P`s`ohwgLR8+*p!Bv(uPiqgA%r7XQemmDw%msq$myT~*!j#arGJ? zUH;~Ih)+R!M}Vjp=j={bu2m<$$550Ik>wk*&9$L#-frpeS!vS3GE@gTY+gZB^>0fO)^mJs2`SJW!`;?PHrr)$5baT%%4Q6>7On_%Xrb=zFzI z`SR^9V;nqlBlc!hQ=Ppyc2>g!f9UqY#;brGlUZ zm=wf*pB6Or9jEgPrzr196;Fn<`GdTkU=|(|14AL!>UvU|*$B2lJ6rJ&UUSI`%S*G((dm7re=04-h$^)qW98>JF>R(x|B!2+4%p>K?PNE)rPuVOc zbur%`)DO-d$*Wu7h7E|gdTgv@*Ax8gtX{X~R>z|~`k+CT!*>&N3HQ?dJ)KL1FYjHd zoS-xQ}k1Or?uL>l@<{$b{6Y8E>^qf z^vPDA1b1+y!oWdH!@79TwuE8_`pofOkQl(*mt*vd=%fIZ z`7HOBS7AzSx3ny>CZHp!$IB};-`^Au3moC%{COl^(9_gR%E}U^kzKmqn5=Z@%#~P@ z1Fy3wWEf#Kz`8PFnT?ek2&LxtG_}4wfEDNn5h^Qyp>_4!3e}mZej;9yszAM6hA_lA zbd)EopMcC{6T^1E!|BQcQJ$*;p-NE|+4!LU2EjIB$(<%?~idI1B?{iU01+tFrh zA0Y(dB~r0tmw+t>79|SmuzQaA@YZk_wx%DL9BdRmDnt<4JZ{3F-?b+U9GDu#dtoN4HmQ#(vBt@>@4bQf;b_5f8sk)*}=XiP7s>d&Q>m1Vv4e-Jlp76zS&c*_0weY26QU5SriD*tiY>^p&*cdPp) z!h)IQAM@n$;fiCWmi~q@?eN0?N(`-o`0hRBtaFsp8!Ume26C zpw8g%8%??+)(Nh$jKM6bVH&BSYF*ZcAet4QnH1B?9MQ(pl!c=LigsgSG7%cDnEd8n zuzyQ`g;~^}itU6r!*0jig%T0%t1Xbg!`EZ|lm5Q|=JXe@h(s_7hUIs^kVJg{`3n;$ Mqas}`X%h550Q;ID$^ZZW literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/liveperson/index.html b/content/ko/case-studies/liveperson/index.html new file mode 100644 index 0000000000..0cadb0f274 --- /dev/null +++ b/content/ko/case-studies/liveperson/index.html @@ -0,0 +1,4 @@ +--- +title: LivePerson +content_url: https://www.openstack.org/videos/video/running-kubernetes-on-openstack-at-liveperson +--- \ No newline at end of file diff --git a/content/ko/case-studies/liveperson/liveperson_logo.png b/content/ko/case-studies/liveperson/liveperson_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..b7e63d94f72d697d0e88e931aee15dc3f0797255 GIT binary patch literal 19458 zcmeI4c|4SD+sChwHH1`DmWe?NX6%eTgk*_Ai)G9VO${^5U_#lFDTTaMxFyPtWU_kG^~W7y$s-dDnkHK>Q7U08p039I+-^8y|uran8~R z6wU!HO>)Ll)BvESNx~zLXV65j1KJ7WqAoH~S}p>{pwvaI6pUfUcpda9jDeRM+QQ4^ z2-52eQUxWVsj*Rw1g9i$MiUWWk~7wY04J%7e9a4|#Ouva5%56ebUqQIV37gHxXJF!1+>h{i@rq~?Zl zgq!Og_+F0ko4Uv;A`uUVLOne_r9I`Oac)i&yNU`FCJU97m7*x52xnc02$GZwLG)Xa zKlA9K2}m~#o`}J@fYXJ39(RC<%dw%1FbYzcZnb zKRkGMH|*C#Lm{DPEZQ0ELL^XpGQaiVPvM9-!YSN8l>E{CkA*2?Vr=}Q_HU2Z+4;An z2}C^)3WD!Qf9oV1Ig3X_&Cvv$yBiX%=Rr9Y(chgt5##uCaQ>2Mz2_f~9ZkaggKfR% zo9*jV_%=dnlyimaxS7I@vvH&|hso z$NXnTv@U{()>xmDQZNp{(H{~xmjBzN8|bU7XaQKO3o(KV}wO2ZzPEQ4R~O zA*TlYT{D&I>uiG?x)6v67bMzHSA)_ZjlrPciZEGu2U$lsDLDlMOiEr!URg@TK?zM+ z^^P(~M+HTglAPjKh{nd$_P?dp#Ub6-SIqa+sQ)lEmHlt2P2Dh*-421J(ySl-e_G-< zY5rEq07Ib6(X&)@mokID?q?Qg*T1&@*@4A;-IwqPHv)Qn)v1g8x$J&yWIy()b7fnRe0y-{g8Z=`%E}Dp_1$120G-$?jTr?3;3+TA0Y0!-6xM(7x7SM4~ z)1VpCanVFXEuiC~ra?2N9}Yjq889`QPZFq({a&6 zL@l7>qNYJJrsJZCh+06$MNNZd{4a5B{CN`~+J$oGpC{!OzX6%LVan}+V5EV$F#vdp z0f1j104%Oi-bVny9R>iS_5c8n1pq!=gx%8v0KjWvsH=U1)b%OR+voc0xLV(GLT?-- zGJGit0@=m7dq5{FOz98OW86BB{w|s^BK!r)v3Fi=srHZ`B$AQ!^vZ(I8i6!aHlZ@)-8gfD(0k=d z?R&H9Tw8A&C*KLWoU(22>02q3;P{>0lwjLlO7Q>L?1P%@%?S{*pLMPi4DRsCF5M;q zfVDnH?OugAOvqk`+WLD`=rct`j8=zT44QJr+I}in4y#s*z1_Rn8f(YM0^qeeM=B)H z4Nj1PxT8tPfs>c6ywq5&9a^#A2Drx8SidoJErVUi%vy@}$Quk|2^>}xd0d~s$j|*C8Kl@3GT7c}<0lQ?cnLI?&@p=g zP6lyStJEa`eu-K6avG1{t-gH`qG+B!StR>Oi zD7!GmQ2WSZ8>@MHzy(%tDX(q({%xM3nx`4}H?!3Agm&$7@4R#N-Ku0_?#E-q9p5HQ4>hXkwmbBE&PDFgbme+JG%F9{vb_yI z-(0K5GP5|JNfMgPT{vIZ;y)CWJ8oAzUUMX=WKf_z3bG5pKn!46RRU+(Ya8p9d+s&W z=h?*+L`~-E3A_b|-Xdp?rZNene$X{JCR@!*#5p>|$T1glrp8P$RC`{Md6C7UGqQ0> zaQO6QepHM|R;twr$QwurXFqqQZV{~Ki*)#~cnNBBA=r;hj(J*E^7tC$v4z7MHh-DP zjog{fju0dgCt9AGSKb)GLn5qnpt)Arp2HowF~B_%&!A;$YbN` za%f2Y^U(b6*Ni(6E&SKzH(4$ew7(rlM!nDyJy;Uo*>fr7Fl;WcAiU19&`0)tgF^-X zvN)G`)V)Y)|LlW=Q60%7MQff!W0#Rdm)ec|Ve%q=GsX{*u5bqY^Ss?RGz>uFPiGd! zkr4@JobF%ybjW74U$64ZQ!*-!*Ge|$v6D)E*UA7M;&=Dhjj0UVc1_hc&dZ|vVG>ZC zx>rI`C(v~ezB3?uMiYQ#@Mx!;U1T%a*3S2ROl2-kB>$jXw4R0XXJ*T!S0Ce(->j_i zcNWJp_k$VWTK0{k00oIpugh%em@ESCbGGq!@VvMu6MXIxZ#m;j+ykadZztMW_Td6e zfk_dMF0N0rxQYw1@&=DoNbd^l8z#7MVt|Yva^#5W*j^!z^EC%H+4?GzMr+>{bETxp zyk}aCm|)70Vzpb`OEA)$BzLby7|LhtXmJPRb-^%hVfneDwVJmZ8J=Y+%1uOSHbsm# z`011RMQ!94wY|#5eBAKtFGV9A$wwKypKjTs%n{{OU!&hSo%-mH#wxZE-(ga{^AazU zxy+fc06@2uAOzX~X9bH(=)`W`Rf#E-H6Q4*tbgNK|30a}N#pAD?K$Zy+5%_`mBk&5 z{=3-71WAqT3Axf(E7=|a*LFEM7%q-U~?Z+>^*k#gXqks^qdY0UuNF%QkO}$ zNL4F|Z4I1F$H@IHamQl>Jf3N2v8T;iC{HWATt4*xeNERY{JxV?^R2m)uaXK()3zwM zbw^E)mRF?9%SKEFDP))PKOIW8e{Qx@ObEEV{iGK7dY4R}PR{^o;&8RUCtu2B3Gu38 zcYQ3GN#D2Fis#~{O&Zpo1&h`BA||@S=#ZSJ3{P%Erlt5E*hSNei|thXhLBVxa;s{U&4?)v8s-4x$4Zeyr`WpSW`C-e_9DeQ({ ziP*7rQT$U|57;;7I{NeUbPHL5v-mEr^`_h;{Rh6I&ifr-@dbs4a(2vF0Ugtju6B0+ z!*+gkjj&O`aC|>jHKl>-*Ry!saXD;_6qrpqv}?0}Sp4+j9b1ajitn_J&#ZID?px$UO3c)q?(bHf22;iDPx+vh(|z(A?)Vv&ihe>B`H(eqGvgOvVw zA~t59aab`+tq$;z1K_{;*=dVFYw2m-TqAg|9xZY77Ku_MV zCTdSVCOnB*=6TBLeKU&RQC9=Z&er@f6WR5=2azgaHZG@_Fyg3`xu-IfDOYD8DAPTd zBmZ9f9@Nm0T*sSbt}*h^fx%kJcEo+>arm{n96T({evI~IJ)p@-U88hkpUXg#&1U6h zQ0L-Y(VI?KDb6(_yzICIVG}^!wVUAryO*ZSl^8zkS<7vLOzoWSJL|0Ck@uTkc^w;! zY$tT96^^!7Ca?!6jpy^O)~UXeh&Wm_bn~Gl3Fdt+qF9}9FwOZ7f?-}v(1`dp{fa9p zVjru5c|!&ve72u#md%=?xf9J$*88q+$TidAOAF+N2xxQ5<3T6g%8fpDvj%KoWvYRx zdta(vs?J;m#ps+WX>T50`}_rJzhYzH@ob+JxYwAA{~Ts8)|5r~j(QL>Cztyga10&C z6$uC}5)$5kZ57!iU>Oh{*E;_CjG5sR%sr0c0Yy@(4Er%Qh^8+a_J!@pPEQN&ic?C! z^hL*vPsa3tb{0tVJT6_`yeE*W+JmEor=?-?KYX&k#3r1k7v=!mcVI`}Yg2ZV~U+g8*6#x2`R0!+0Y#FOT-_an6~?rZci`&R$Z? zDEBiAX}IMndIWw4<*2-KNiwcx`u#P=l!T|+d!15HUEm>War5A7637pdmcEzMEadKq z165bl;_@@%?{43|rTg5wX5`TuM)uXaRXTF1)qwvzdqg|GGJHr9?r9mXMdm~OI}4SzFb=~lVe`Z~buCYXP4@j2-D z?g2i7MU@A|tt%q2&->WCW(C_cON8@AOT%uTiR3q}25nQhOwsdV!3RHIhIrn3&?4ZSmwlo? zy6^hClEF0sv>Vd%heUCe*Fs{y#N4aowIN)erOL_HQ}sBB|I{%r&wHEYmqBfwjBrY}E1)Zc$k3f8cJW2CwL8F}7_A557F| z&pp^&sg?5a;UXK>_Hu(!+TwO{*p>Q7Z8Umsu;NlU>t{*zhH7ZS6k&Owy}l;x+`huG zt!q%t>q^2MGl#RqNJBCfaKUS~at~F$oVJi}TYG|?Z$qv;IXv0X-W{*k`%ts$878hU z>C3s(2bwsVyq_tQkN2LuU^WgLs@h;9>J+oT*&Mi^4MlBc}u50bb~>P$VT ztF>OIXQ;(3@X8kvmRK_o3tt6bZ2J77gjoKex+aAomvFQ57daCv`J5Q-9}?2C%Q%xO znt4R9YQl^8=RDY3EUv4pT6TVMm~1y1D{gx5;Cz$9d+7&f#yMB#q(-ocw@q}3C#IjC zNJA#7Pi^YT?%m1rdho?kPeeU$NYhNsjXGDpj;g~EJ$w9inN72a(jb;}Rg)1tOiD$yfd4q1t z`KtH6i#})9yU3}YlX0B+P;A+3%iRxx0j+r#Nf*tw=do9NjbNVfrJ6>1=T4ik<;`5P zdG0?X)OM0wxW?{IV416t%{Y$AiLB6qc*7tW@9c%AndJwM8|YZI$> zRsZI6hrRQ;W(IZlbJd|IVCF8S@xp4Zm&$xZO_!hE%#@d}x^Y%>buQ8VX+!07QY#P? z`NT>6#YGF*L(2oaFS^Bi1{&X9T`HQ*)zac;8}g2-oqRddIZ>3BVSemImc4SsqMd#c zH+-$IPU1dKpKxgK6HG%SR}JJ7q~?pBhlDF^ zeqI|GIMr^t^St=S942r>xXruSB)B0s(^%KA`SXXZnZ|Rk%uLET*j=NKf%KV~wFU%2 zS4^j-^N|{s9P>|+1!|#%ImY)ak8K*1c)J;|KCzuFs`e0J9oBaHVRBDoVdZ7^LmpK} zm@>mUA8}7Kw6FiI%(X{_jkC{mKygdQrKG9QQubZ`ydpPD(il^R%x8op0iF;b{6d^h zzRLK4_}M;kiksr;`_1b{Pg#yzQ(jlTfCS9sb!z&=#LlMgg^8K$cO?f-gv`A25#?ra zWpmZ|1H#U0%X!Ffv%8gHgVI&5BR8PD<84W1tz~$A>?-F$6a69pDtt9m!9kma&DgOe zZG6o4K#JVwUjOaq@)^$heV2O0;H}>pwok_iXy6r_?$7+4<%L2MZy0}{8KdqtrVYP@c;$CK7hQHSSphrB;o&$mTW`{L~%gheH+OH?2 zcBhyjv*-4<^UsEJHr)Mhe24UZ`s!&dc6kj5##*O;5WiNXCA$L`6U#6;P3v(epPy{(8yibE$z4`tCzjIE4uo3BGgImiBS3v@2EmiWzKlt17x64u^$Qe>9>eb z)MMLGWQhM~sXJ5C#b3S;-mgq?ni9@w4kZ;SjYMs!5&=@aabqa`N_Tp1b&;$47fx?M z&aXY<&51-@OjPR*g%x3q#I30e*W_ey^HGsdgs(&Em=XAA+3mdlEneOCOr+)Ksf52E z?u^(T;s?pu>wx0e{fXraMI3umX(#{EDlhIDTKsv>-S2lNZ^{U1wATp>J!gG>dYZeb zi(Kwv=f+@vF(HHTFtQOda^G0oGy(lJQJD6WI@<}b!0WHLw*PE0F~$ZDk@zGG9z2*W zvkS-bJ+;KMHi(6_4MQy7e&&xf0x#9Jd(QP77T1ciKtpWH^eEnR>~YxMeAsQcgtCd zWdEw6twTXM3jF-4V~8~DA?zYEn)UTa39|SY7Nn02mR%lUPIv{ft$ToB@_Y}#JyP)V zgoY%FXMWHR2vXJOIsop$ve_}MNZ=32haiSy3HPYmrwTrF`^ zaDtT};_j^`qTL>9Oyn8_nR)z3Tt6u)Y1+NlA^i+T5}dh#x0O+eHS1fg_Bcz`uGc3f zUEU+z2S&%h3kf5OSv!>M=R^FSJfBF8SAtUxfs&YH=b>4JBZ<_?GBSe2%^c-hAP0uz zKX4P@>2K|PX}OtoA@CCI;JA|z&;jsKs<{jyp*tDM1oqXeE&BKiFK13NLR6qaQRH5q zn8NNy_%Tpx%sGdm6u@HHFrr$QGok}w+o1=Z+OsWqM>y4*$_w_~>06GqO7QO|DGbw` zOaHq-@GVP*i)(U$Lt-+dZvA>1Lq_+o7PaX6gUHkeqQ77LSq?3QamGDft%8yHPrs*o zZ`p`1A|b~gUYuI2?1lO#atA(_j#PfG8yyNg8gvXiDGi0j6a?e?urOT6vGf(l(BJ+_ zvFD81yP-DLO0+A^)jpZ+hrfE7xEm;q#$FZUXZ2GfOhM zXd%vO+ZI3X&1U{DAEKeI^`Qtl3*+w_vfxT}Xi2gO7<>=CupZde!DhOG&Hg8rxgm zo!y^R8C_64pbU+SjL`GvbGi^G~o@%gVq(xp# z=*yA}vf{b|ZBR$QaD;jD!*A(vYVp!;pl~|g2U}eSs4d3PZ(q^;pd}XDw2`Gjh^YRx z76ByiQHgB@Gzh7RU|~lbMs>SciFafDZdte*93-U%s;q{iSxe(+Ak4i7oMYPXi@;i-+t164z2kG)SJmwxs78myp>-*Cryo(aD zp44giH#>Bn7C+~_z#%#W(rtmJc-H+Qy`tyhNk7AG!yQ`nMJ% z&7%7RX%=OeruO zyJOqdCjC)i9@*4Qh5=NX(m)TIW_Xf3X4|`6568wTP=-gPTJC>ILG$^%R52i7rc~~e zK0k+Q%m-f2QPWYY^A7NUJMQtyXKMKO-0CHJ)Eg8YjUO$sfReIzrGXfR9p3xA+fAR0 zT3omm6A_0BcP_^RKZwMZv06fNK#ul1j`X{z&olFO&B7&!pl8g_{j9 zno`IrfF^#Ke5MW^mEYBZ#D6$Ds);({d3k{yl_%83=D!Ym7)p-U2+G0l_}2lYWLCN6 z$(p+aT;$MaA5K<5JEB#OM~W&c?c5~yV2l}KT9u?+(bm$*yrp-HhN`5~PfGdve~7EY z`y=}+T;EZv^UMxkP_o87Ty=>c)4h*hmOgLf_~jqBhdz zfN{>}y|t`A{EkNi9ns4{8!^Qi$EXS`N!)mx-=%qEvJAL*Wf>^~y$?%TPt3%0GO3oQ zIpqSI)OWmqMCk}>+&l@i60yUH)LNcNaM%+F*3+90srp(KOgz_}oh;yYV?#O9D|Q`! zQY8y12od?2f8Ii9i>A@;EnCSm`lNmQZf3sAA+#Va?^fq_?i47|+TR zGt+dd3yFKbV5pv=ekgXKApkN)Cq$wUa-x&y4Wf@polnp)mpYal+*}H#hNNMkjpPeD zet4d`Ku;tpDo7sEPNwIZTbe8pCe)Fy>upYm#!WC7{t~hHI`j(6%#do!8GsNJlUQ-+ zZ9O@{nHh)Utkr(Bq_lonFNQfXonf$NN=P@AC;J?XEz{PN-{VW^D((pRb|t}jA5%ew zqYRwSM7R`uxdU@MsSkU8-GM%iQ^P)tKUrialBS7S`S_Oqm=uR%|I;W(^(w2XpF=sm zbDH(z=ku-Xpn>@^G%j>|aJKLgsjZb=m;(vDaBp6cE1StxlGvhR3sw1WqV)S{4*$C` zd>q3XTc*2yls@8TwnW4+T{kyNYggNNm$LP;9YMit8J_Fv@DZ0%o~>9&rq@iaZNkJ) zY#!0KGxEi6Nbaa5Pz)2h?~HslzzJgX%r0@p1%w za5%F=V?gx?OFUmJH1o5_Dxk|)5GYsAn(I*jZ9yEiw1x0KFviHD3k@5PchseXfG;Ty zGg0cw?_$Bnx$&vZr20Rtu6Z=ZF6r&IfKNbojW|!Ti9|hJ-%O>4ajFyMMgf zkM?^j)o6$~nUptb{I5cs;H~M}I^#EH=%cYz5%Lr##;&$Uv50d;0SwFy1Ba&@+xw8+ zEhNLp{>W~xJ*5BC#_6e+W1Co?(Vp&A+8@fBR(xiawDX|x#G9~K^5G`BQQ`oW{-J)~qNnz<$Vnjw+s)&gm7 z?H+AjgSc7f{3yVB>d}1mg!!^e!`H79g!NM$FBQF>ksnHNsK? z)t?jT8@%Vr-(Kz8{wwctf_AM0pJt#GAT2{fmhSV5ZH=D=n674qb@*?+@YcRjkg26= zAIz~!xUwXkoe4lNNPa4tktfRUst~z!UAXeJ4%H?z2;)qH9O9Sti6&P3_UmBl_sP>G zzbMHkSU$Dk4&DMn&Q^a`Ouv9>NvJ4anmO=B` zak*u0Fs!2@B2~tkuQD27@j#0h(4}X}noF4<-?}zakk%W~{9$0(h6&bR6>UkBIQ3E( zLLKLEq@2z|Ye{DFk{TvX8^Nkk+=Q>Gh9J`Gxl5)$ab#D&x^37w=)sfNd|Z%<7yZsy zhCX@fQrzax@BvueY?0iit4)}#HKn+(?my=L92~TnCl7=5UA~_8FV1x;+3Wwq`w&z^ zLkm$;RsFg8h)2bEA`|!8d74+?EH7KEKQ=s^fBS1|tI@~~3cJqHef3>snX=+fN3=v; zfUtB4%nT^T{fcYa3uVw2TGtx(vRoCg?+5*roCqHQpM;!YVtJ~l_DPF1cfuBCr<~nF zWe+7EbXLa&WLimYyJ>QS-nM|Bzd3k}=DK?B|AHNQbKQjq`n~VL4EUTdE{8 zRKzzu5h2#Dbz)ow7aP8WL z4?Xx-ya9CLx!gV1UMu-$sY&Kb+w`IrYZYMWTlvZN*!q7=P%hJWquZBk(OXVQ#kx&C zN0Pe3kQV`s)z`*AvDniFC96tBpWVlVwH-fJ00O>Xg#?gW2+50xtgZ!Y;C_mh{C;yk zfL)o0&L=-7M`JhOIoqYJIv9RL1vA&=>`oT-ad1xCLp^OTEwvYABvso8Ck9Pc3znCd zdao@YZ{~@Bq?#T3kMl>YJSQh1407_>Mn|FYTxH(5M~h>xKC8Wtd05;UJtO7b$Zv#^vIy1Ams6j3Tt(9gz*iGceMGT%oClD zg2j4L%9))X>v9Xb|GI#~U%Gif4zA-Qj3m`cNL7-z?{e8Q7p&jjx=7ed@Ykwx(0WTd z0*WAkkvy59wxuMn=740)s^m86uT@HZ*c~koaB?Jz~i=WyLT|ct90L)}N3+}8r zB>v=+KldlvAA&9o+$Er7SI;Ne+l~|Lni)2%K1J%L|A^1m3Vm$&DCYoiND&a08bV~Do_=$cq#B$YJ`F8S(+kJrm?k4d-M>l!l0q!I z^8sNs%kk{wvk$kQix7i5DtT*EEWEQ8zk5cX&*Rj|+>TRfb_&%NFV4FCz3e2<=oBmr zO{9VeISEIG6iV5dHHGl4)oj%oHATJ|VA4~r-i4b_d)PH%;}1v3SAya^6& zLA|8=uE-qFf+IHdL0s1G%%nh+3x9mZpYh+qH@>$+E^emVCRl1~Qxz&z6fmo(x|gq2 zhKmE^Kfxj19XzA~M@PBOJZX~%rZicOuXoJPhnekIs0O4S0ji~2Tm5WS))n`^gGuI;KC>CLcTAI!p*4(! zI4@C9u5`S!bqWfp9?gOT3L8W*Iw&~HGJN1pfsIpqlmolSyaU89vq!MjFdj8LT=Qe0 z%HSIhq0-e0l_!?{bTQTv#a4U797Bd|UBVw`pGoFEiDwkj7A%OO+>l{^F?|$F^JNsL zh#JE7zE%zO{#2|*aJ9v!gcM$zZ#(5D*#T-L&WcMSTfBig22|9@9qMtgjIg~rWxI@0-fxtjws@#gT zhEIfIX8Qdpg0ru{e3QV#YNXZM1yz&A6!3z}z*?w{T=l&3zVZHyQCg + Our culture and technology transition is a strategy embraced by our top leaders. It has already proven successful by allowing us to accelerate our value pipeline by more than double while decreasing our costs by more than half. +--- + +
+

CASE STUDY:
A Culture and Technology Transition Enabled by Kubernetes

+ +
+ +
+ Company  National Association of Insurance Commissioners (NAIC)     Location  Washington, DC     Industry  Regulatory +
+ +
+
+
+
+

Challenge

+ The National Association of Insurance Commissioners (NAIC), the U.S. standard-setting and regulatory support organization, was looking for a way to deliver new services faster to provide more value for members and staff. It also needed greater agility to improve productivity internally. +

+

Solution

+ Beginning in 2016, they started using Cloud Native Computing Foundation (CNCF) tools such as Prometheus. NAIC began hosting internal systems and development systems on Kubernetes at the beginning of 2018, as part of a broad move toward the public cloud. "Our culture and technology transition is a strategy embraced by our top leaders," says Dan Barker, Chief Enterprise Architect. "It has already proven successful by allowing us to accelerate our value pipeline by more than double while decreasing our costs by more than half. We are also seeing customer satisfaction increase as we add more and more applications to these new technologies." + + + +
+ +
+ + +

Impact

+ Leveraging Kubernetes, "our development teams can create rapid prototypes far faster than they used to," Barker said. Applications running on Kubernetes are more resilient than those running in other environments. The deployment of open source solutions is helping influence company culture, as NAIC becomes a more open and transparent organization. +

+ "We completed a small prototype in two days that would have previously taken at least a month," Barker says. Resiliency is currently measured in how much downtime systems have. "They’ve basically had none, and the occasional issue is remedied in minutes," he says. + + +
+ +
+
+
+
+ "Our culture and technology transition is a strategy embraced by our top leaders. It has already proven successful by allowing us to accelerate our value pipeline by more than double while decreasing our costs by more than half. We are also seeing customer satisfaction increase as we add more and more applications to these new technologies."

- Dan Barker, Chief Enterprise Architect, NAIC +
+
+
+
+ NAIC—which was created and overseen by the chief insurance regulators from the 50 states, the District of Columbia and five U.S. territories—provides a means through which state insurance regulators establish standards and best practices, conduct peer reviews, and coordinate their regulatory oversight. Their staff supports these efforts and represents the collective views of regulators in the United States and internationally. NAIC members, together with the organization’s central resources, form the national system of state-based insurance regulation in the United States.

+The organization has been using the cloud for years, and wanted to find more ways to quickly deliver new services that provide more value for members and staff. They looked to Kubernetes for a solution. Within NAIC, several groups are leveraging Kubernetes, one being the Platform Engineering Team. "The team building out these tools are not only deploying and operating Kubernetes, but they’re also using them," Barker says. "In fact, we’re using GitLab to deploy Kubernetes with a pipeline using kops. This team was created from developers, operators, and quality engineers from across the company, so their jobs have changed quite a bit."

+In addition, NAIC is onboarding teams to the new platform, and those teams have seen a lot of change in how they work and what they can do. "They now have more power in creating their own infrastructure and deploying their own applications," Barker says. They also use pipelines to facilitate their currently manual processes. NAIC has consumers who are using GitLab heavily, and they’re starting to use Kubernetes to deploy simple applications that help their internal processes. + + +
+
+
+
+ "In our experience, vendor lock-in and tooling that is highly specific results in less resilient technology with fewer minds working to solve problems and grow the community."

- Dan Barker, Chief Enterprise Architect, NAIC +
+
+
+
+ "We needed greater agility to enable our own productivity internally," he says. "We decided it was right for us to move everything to the public cloud [Amazon Web Services] to help with that process and be able to access many of the native tools that allows us to move faster by not needing to build everything." +The NAIC also wanted to be cloud-agnostic, "and Kubernetes helps with this for our compute layer," Barker says. "Compute is pretty standard across the clouds, and now we can take advantage of any of them while getting all of the other features Kubernetes offers."

+The NAIC currently hosts internal systems and development systems on Kubernetes, and has already seen how impactful it can be. "Our development teams can create rapid prototypes in minutes instead of weeks," Barker says. "This recently happened with an internal tool that had no measurable wait time on the infrastructure. It was solely development bound. There is now a central shared resource that lives in AWS, which means it can grow as needed." +The native integrations into Kubernetes at NAIC has made it easy to write code and have it running in minutes instead of weeks. Applications running on Kubernetes have also proven to be more resilient than those running in other environments. "We even have teams using this to create more internal tools to help with communication or automating some of their current tasks," Barker says. +

+"We knew that Kubernetes had become the de facto standard for container orchestration," he says. "Two major factors for selecting this were the three major cloud vendors hosting their own versions and having it hosted in a neutral party as fully open source." +

+As for other CNCF projects, NAIC is using Prometheus on a small scale and hopes to continue using it moving forward because of the seamless integration with Kubernetes. The Association also is considering gRPC as its internal communications standard, Envoy in conjunction with Istio for service mesh, OpenTracing and Jaeger for tracing aggregation, and Fluentd with its Elasticsearch cluster. + +
+
+
+
+ "We knew that Kubernetes had become the de facto standard for container orchestration. Two major factors for selecting this were the three major cloud vendors hosting their own versions and having it hosted in a neutral party as fully open source."

- Dan Barker, Chief Enterprise Architect, NAIC +
+
+ + +
+
+ +The open governance and broad industry participation in CNCF provided a comfort level with the technology, Barker says. "We also see it as helping to influence our own company culture," he says. "We’re moving to be a more open and transparent company, and we are encouraging our staff to get involved with the different working groups and codebases. We recently became CNCF members to help further our commitment to community contribution and transparency."

+Factors such as vendor-neutrality and cross-industry investment were important in the selection. "In our experience, vendor lock-in and tooling that is highly specific results in less resilient technology with fewer minds working to solve problems and grow the community," Barker says.

+NAIC is a largely Oracle shop, Barker says, and has been running mostly Java on JBoss. "However, we have years of history with other applications," he says. "Some of these have been migrated by completely rewriting the application, while others are just being modified slightly to fit into this new paradigm."

+Running on AWS cloud, the Association has not specifically taken a microservices approach. "We are moving to microservices where practical, but we haven’t found that it’s a necessity to operate them within Kubernetes," Barker says

+All of its databases are currently running within public cloud services, but they have explored eventually running those in Kubernetes, as it makes sense. "We’re doing this to get more reuse from common components and to limit our failure domains to something more manageable and observable," Barker says. + + +
+ +
+
+ "We have been able to move much faster at lower cost than we were able to in the past," Barker says. "We were able to complete one of our projects in a year, when the previous version took over two years. And the new project cost $500,000 while the original required $3 million, and with fewer defects. We are also able to push out new features much faster."

- Dan Barker, Chief Enterprise Architect, NAIC + +
+
+
+ +NAIC has seen a significant business impact from its efforts. "We have been able to move much faster at lower cost than we were able to in the past," Barker says. "We were able to complete one of our projects in a year, when the previous version took over two years. And the new project cost $500,000 while the original required $3 million, and with fewer defects. We are also able to push out new features much faster." +He says the organization is moving toward continuous deployment "because the business case makes sense. The research is becoming very hard to argue with. We want to reduce our batch sizes and optimize on delivering value to customers and not feature count. This is requiring a larger cultural shift than just a technology shift." +NAIC is "becoming more open and transparent, as well as more resilient to failure," Barker says. "Even our customers are wanting more and more of this and trying to figure out how they can work with us to accomplish our mutual goals faster. Members of the insurance industry have reached out so that we can better learn together and grow as an industry." + +
+ +
diff --git a/content/ko/case-studies/naic/naic_featured_logo.png b/content/ko/case-studies/naic/naic_featured_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..f2497114bf40a1a0fad1eaf65fb52b2c03918f08 GIT binary patch literal 9433 zcmbt)byOVBvoG!hTb$q?g1fUg1Y0!t;;xJP;_eax1cwa?kPw19A&}sb0KpxCTX5dw zdw;)s?>q0l_s`pNW@n~ORZn$Sb$8XLI$m2-86SrV2MGxYUsXi`0^ENBR~Qxw@PDS= z4FztLFhwJnuDcz~59(!$BxmDpWlOK>3bnU|*g|arJcn#0k&sZ?9rcW0Mj9`~t=(O@ zp?`U}{arl(X(S{`8GjF`wX-dZ-pbbA(M^i+sI7~U-qA*i(NIW(SHnZz*1=IF(92dg zP*cx3(AiqdhEYbEUeaG2Xu#DL2Br6Rb#e0+_m^V)r(JR2`7fD=k^UbRn6nh)e<)?7 zp-nIE?qy3a#4Y^Xnop3IUPO$WPncIwh>we&pO;UBhgXP)SNJ(EpE$3mI6oi#zkZCs z*Su`(#32ew|N0j2CdKFggL#Pa@c8-par+5yyL;L5@QI0u@$mBV@bf)5B1>T z6GyZd-q1NN|E`nNDJcK`1W{bdBS5m#`x z_Hng!gQ+S=F#;5B8%G;)UO}*uyn?&{m`|9Wk55rlOjtlpQACa(tSBZA782$Cx2&q0 zHw@}#ZToLo$N!S$|9{Gg%X`^EVeVdf?(Q!Cww|_wJIvkN!QF#iUj83z(X(qntsUL| z5;^`3@W1r6^>XyJwNdhNccuSlgv1^H2QBz{6~#mqg!#qfgarTT0l%n#qJWURl9G^s zAXos*_%B(T|Ec$iyov%MO1xrnA_D)TEDxXz&tLQYH*^1+1o+V3$NzL*;N?Hv-_{NA zc`v}ZNxDjbS;nBOsvxK5KfiB==?_x8ebbR|9u86g)60Uf(9*_VpKF(MV8w+CIzpu7 zu;bt=M4-C|Ms|v~5Of_-JrYhZ2%QK7#$_R#3~KwHegAmLT43+hv|z>^vD2(Phw-lX`SnG>wZeNg&P8iw2KKJm2jN-w^u@(RN<0e6AzcpJPzENlLf-$b zPzvMHh87yXQ_%*cnny|i*Xx5@zs#aEt0GV)v5G1!N}`BqsgWKMYk`NVm1yQjTcEe= zZ5(5K=SQou7v{5&uSg( z`E6h_X+(qTFJS@cRP)E1-*rFVy%gZ$8h%&#De|+++}i5uD!bCf@1J!B z?~Us^Z=B4{i-a~ygaC6S9SXY4YF{1fIYr_OZ#W04{O~p0Pc*=Ll5R(OX87^kuOZQ_ z-6<+2X0atC=;T*ojEs!Uc)0zsi2rzaL`dKh%5lF{YQA_5$4-XTAx_ocltXiAi}H0* zt*$nUcf|BN#>rPRY|&zy8>sewXdh)$v)fVSNS~X0eo-#_TG(~8NXn$BDERUIk5u3} z??aSq6VL~4?$LF667i_v@rPBzVgYQwrBl~<9N1SwQPoJ>sXb-MTx znJ3h$ZV9tNHt{{^dg(Qn%0)X+&4Qrf`sy|8c6+# zP6LONR2!DtU!_%Zb7l?MAR{ANDKgq^%Q82==dO2Q z5&k(^PEBX8LJ?ME2-f`yCR}*DI~#izC!?yWn#rccjqAyT9DF(JCd+GajFJn}X`eL0 z_;Wb+LTZgbR+}?BCMM?LxR0RjMPWvM{^`%>JtHa|!sP~{I<3Uc68s=&q9v7&q>DR96KV32ypPzK) z*72~j50Ho!6FeI(e1Yuw5Tf=i!HjtFT!!(S|NFmr8Hrk6W2A3Pb-WE2}> zK9de0RBmRR^xfIN%BK`V?{w3A`SK;l&=7{%jyetBnfBa~=H%3rV(sBv!`H+7#f1fj zZ_hs!TYJ;#xFVXH83*U)<}$S4v@BCPO`&>c{uQ`Qb*UdU@DII&wP;6`oCWja;^O3E zI}3%@8PH0u?LstKUx1Kuy)t(iqi%1X5ov-~_`O~R?B0yI@51jE+y}naIEY5Xylc+r#UVfaMoT;&v2g#CC}ssKK~MdvFFue_zg)Wc zQTx1x%F6k9 z5NLU|ODMiT$C#Ly5M}ps_HW`rzna|NJk1tynvSCt6&76T2=d$HRkq0=h!Nw8l$foI zfVo-kZgO=X9A|c6SP6cIv!SOO@fo_PWNEA&Dr>fkAGlTYJ;W4kHql4oT_QP8YFYXR z^Gi8ndNm8hS08t$s=lVu@j$qknKugkmOeC|vTEe7ggH4mK_D4rRKQYi$HT)z1De07 z9rksa`&cn|=IH3y_4v@%*jVlVN}{PeTCyq+wEIxGU>SXScD8JC?Kb3&#yHWS0x7_c z=R-=&YWEer2>)1itb3@?9uC$gm52WN?24eSHK^GhdCew$ZFm~kEq;1(;<(f<#@9aY zHSJ7%MIgv+-e5PfH&gR0OJJtPFxJB1auV2u>>`zoWLbr>8_%*AhR7Fsy54;%XRxCnt7YZ&5o90w#u8%!od;0Sx-GVIq7vLpqt!f$q4*D+RwW>lBk*YE{Ef3BTD0` z_H2u{{aqJqH{)bDnObDYe;q&FWGVP;1X!qk7lOy@x*m)lDlhkD5^HTrHvZt%GJke-0Z%^S1!!l=T(+#gVP%w?8YFdi&?dU5}0Q8;!c= zaiN5t8Yqv{Q?)8TgLiB)fpBAOFF`HzVK=X5Mmw#tBl-^`F-L$^b%yrN;i2Ys&G-4- z?V=#f4W%8hGX!;)JppxIZ627_O-&+CDm4pln$U7^%2rlZZZ3Ym?m)A3=|DPv%i@#q zDxfTe_i>=n867T#r4Z#im89PmHr_IJ60dTrIZwshfe5K6-7cr3VpxwJhjo zP{~AcTkbR)_4Peky06GJ@9k;v-u_W*X7WlG({8sn8rLA-x}vJQB9*yf@WtToy|uv0 z{l0~iRvj>Q(H*spLqSFdFiqxa>N+ls)e?++)2^mH_4Ghegj!XNtlYwyRRq++?=nJ2 z8_v(hWK1Y?r~hog7n>!MDT5ihc|H(B#1KF8H!6PdN}Oqdp>4hg)izZX6|bIKo_7yJ zY7E}njTO1>IxV*P?8=EXvMzPB48%h7e~fD*9Z;>JL7@f5cBC)wm9E>w@0uy*)Lu znO#d~Ql88^@A$J+^9dQ)B6GkbD)7d+eMxMOO%Md~BNpc4(-HP)j%{z5QYiH_16}t* z8|z4viwh4k?BTjfFSd;f!#QNpYvQ{c40j=pP zR6=56D~`Uh>0me(8aCJaHcCaTjAtM*F#>Y-o0v4V5tes?ATSXp2#4w-t?Gi=7BzmZ~i}IzhCq zNEX0W>6&uRivwAM7h2#BnOk=dWW~0+RZ8`F_LfLzEJ@g zG%E^g${R^s*wTwSab_Fcl=GEWh?)8BgCfI+L@A4$izRW<2;^$sr_JG~o8mif*Whna zuuq>Ve0$aOoST~)LvT;MO6)F3n}Uc)j=*KUTlT#{;#eN+aN!kIV5{3m3fH5S%suEt z=m{YKItBYU&BR#_Yy3CWoUIyH+|NkQ@!wQzkrJPIW>@7xcXK+8ur!+4h~h}y(jB)< zt*lBLDb-kgDLwnbkx{uwjk-ebWALeWKD-}#4`eaHCtt`B@{S{5>o$AF16&P$0Ra+X zViyl=p$(-cb!qwT0Um{|59WtMa{=YegYzt`kGy6o78&^=A$KZN{5C48ta0expPgnn z1G%K>%#BZN_}6>wpKVuXt5x)gt^fY*Y}#R*F#go!PfJ_jnDd35seIha!j%=3o43Z< z>Kh4RPjOxgI-;19cq0UVq$!nAinuK>B_gk_@tCPi%r`pk-dc&~E{FVCT(=5oup{fr zI-k;#nKQ`EVg?;Y8w=Ey9RG+~pZDY6@Y*gXZfPj2)7lxZpz7Q!Nn_;;-(O<${#k35 zdOqcvUl$Y03++XF%UrD!YRbF@njhw7@e@)H6G;67$%$e3>gkAZTnxG-R=SzQhWpP+ z(<8%;*S3cB6{2)MunV%fOP?+lVqF|9TP#SB}t@W!pHO$Z&PXld!q? z93?hfGsTO`*b6wX0Y22%M~%O`l+qs}kHa7RQeJ{xSElpBzMe%3q>`DtGlp3)WJZ=@ zM4)~PXnAvME5Uu%>0WT;^5EC5csCF;y==)@;c45__VV%qN7-CX0=duofFV7yxVFPR z{+a%{7gs&Nk~eDhK}mLb^?83&tt%4z^!vC-4co7A{m%k(2&{~tpT|xUp3WGYgweIN zwdeX$!3SQIqhALG5}MTvIxmiX*3lJW$A0v*B(4cTN{);3@b+$q0td~n_AUk#%lKQ6 zuZ=ohPb!JAu&^|T&YreR0bwFD+t@tC7hV$8$?G`Fx#(@sGK)%%IX)}f&kQYd=B6={ zqj1jfUh|H?35`!Q8GJyfsoj9PN|XNw~I+AXE3SrI5Z@x`q(D zN$3aHK2~U-P8oXz3Lwl)X7}sq8*qx#Sl<%%4PbB9;VXT6XZG2vrPlNH^x5{R_a+j!vM!I3-KR9n`cqyuJ~qZ9E}l=@gAbD^6n!>}phXtH(Ta~8AGfj(Z&fvYzw&DkPam%)%pT;%#>&>v4 zer`*(dP%KXJ6W}Ty>gXYGh|qYMT-9t>?Fjv>-RR?!hdI+p$yU6^OY`aKh5cTWsfw3 zoZGB*T2SQ2&C^wliyRfrb=MyVEk%sS#7$D)Sd2^~b&_j=bvH6gug#2r=zSq(rtwQ>WN=k6-)T9v!*iJpL zK}q~MK-(}C>s3__)onhkc|)zAd4q}8Lq;vnj{TLW<1jF%QnoJ{E8ETA_(NUnS`FpXr8hWNkG6ktqDOH7CpyUU>GbYlDJ zBwj7SBl$vqg9kTLn;c6BA080OacGtCn6)aLHR*Gb7K}osLU}tA$$19C*0+T0*2e|x zM_=Ue_CSb*@ceEs4*)2DOYvg`*6Nd1-#>?D#qHBd?zFVD0H&erTJf=;PSpeIzR>)^ zV-=jZnp?r6A=|XGsB-tEXRdAQ1;vl%?X>Ch^H0-y?+u*urLZk~L-f?*P2s4Tb=!#- z?UP>?n+%E#YOY<*Dix2y@^bHvD-5TWZ1ypr7U}JUUJ6TX|iqr9~e}meI=vJ@P?Zi$$=ZM5|R0mt72+MR!}o4kL9P_u^s& zrqD_m3vc7>Oi&LV+R4_f`%3t#@S8k$lGndUP~jnc+TpcRj*#AaWDsNR$%`k zT+3)+J2@sH;|jVVs>_E9VLoKww>GSiSX1IW3K>Q9<-buq<&VNL>@MIzEWS`VU&CA8 zjM4b{HrlmTNRj!%Zb&`*n=x&-_7zW5nfYv<@pIz$hL`goY-)0))ItDe*0%+Hbfi+M z%qsgFh8lQ#-@CK3)7uM9S)WCc6n?vyg~q8G(T19NbifoP;)Zi8lz-pw1Av{o{x`fi zkYC&X;Vz)nj*pvLb7P&i*=EXWW8}5S>(D-vvWWtzpdTrlzk!XLQKL_1=1c zsMu|0&Kv@R!Bj#{(+FL*-#-hd-|DL)^r+6RA_hIz2V8-)dS|>uSjc_4T8A(dLTBGk zQG6=yvlkT`>$f}krLs<97bYHFEEkCZzxsrzaQw@R}R9LYjFG+Ci;Kay{2 z!2+a@ZWEEzeuF zi`v2-A1XCo?__ANWQTg@QTQqa9j`=u7$fYL3c4K6+Uo9>wfw0U96uBb3FkD%rfu>& zGH-n+DlNSPn_}8&0iuNaZJEbw086R)`CF?%RL+o*oQzD^eTl}Q@;#M+9o~#;#hW6 zCUBhHxMGVPqD8dd0C-VdZtB?X6co?xl9-%q))9CCM109x;G$yAft07}lpFi|?cUp( zmYx%(ip^9YP`@XQ9_TPRA)!>AVOMJbC3w2dg03uD4G26egTDdWh>ksAW6!$@WxDID zD?D0Bih)nS?gNEFdH-0l+sJZe19{F+(B*0@-5Ny)5|_a{BV_in6EI_^F}~nB zQm!SY-DrVoS21$%3i0A5_4e+R4#5WUu)@f4*rODu9JQ&bX;HuyrIk@_G(Uiip)Oa? z%Gw2ogm@B*hW>uv6iy*o;lA8iO_-#}H*1A;07U%S81~)60A{{=6-{oA6n0GSfguA; zZ*OluZU%O$QYL>!PGz2M!K~Quxn!1Go!Kfax_WHXA9J$j8vu|M2knXiB2rWw(Z~nu z_FNtdb0F6S$~~r10MvX#MnclyzARlvq5TvoJEkp(y}!9PU;eQte2oo**76Io*(|cn z3Pl-~nR@mnfy|w(lzIVXZjZAu!3c5;8c2H7`M@i$=)Eg$r? zn>?_?9}a#s$%{chC9y}ntRkcA$dmBnHi8B+b;t6&qk~ zdTD`dL_q-!M?3Kvh;~+Fro~3bg{D8*wf@BM$0~=5Ws!E1NaDkiKc>5RM=EofHm6B0 zED)6@3InJ!<|DPROWa#+>=_$T3~Nk$s;EvR6YWVoCII@uy@__CkLg5(TlNA_1i_oA zL$2^ArR#(gKG%bxiFky?i)pCr!B~WWDN6D|-#5M3J+KHB5-1Fcdw6@`NC=oBf#{Pt z#U_S2oM@DrBVZC>v_-I8nOCYOcS|ocyUg)>@G>wUbH$wY?SL)d5<4lum83x7>uOr~FMsZKlLaB5?4*aH54VyP?l=KJ4P?x>`ZY0ckZ9%!M(_tq zX?IXyltp;73DglX=R{k$AJMvc^Ich=_4AZJOWL8m1_}emoYgg$m1TYPqObsJe|+w; zFVmh8_L1fh1Ad3St~a@&eWe6OF$Rnqolh^E(h{tVoollk;i_P#2t>kgR&>br}tzU3@f?qvH3QX z@>HT!*eD6|jNln|^vXH_K=0BD9&6ImQG`dJViNbo9(f?OL^wr7$t5#Xqa~Jb(wj7+ zWn=sZzqQ0!Y(wRY{YWZXLWYb69R09NMN*(Aw(~sQ7_wxHp@|TC0FX93X{2Fx8**d( ze2gbJXrNJ?W`dYfeOUtkSf>bLJWj!$i85vUW%hQNpi7rH{L*LaNY!W#SROPY5RdUj zZy4NW+z~yrh1Z3Yqs;_J+*iuGHXmWLc7943fZFiS>6*YX353uXNC}KaBQS7Ej$Wm z0==F*70D3MU0hgi=+Dh1bZ4pZv>ttoz#p8+tAr&n9{P!d31u*aMvQn2ZeDB@RT_OH z>ba({fa`5LGbZ^%%&HaMzf46zFg&&-I-8jw=X~@VR*J;ceK(?E8RP>aSYGcoBjrEnSoFMWt0{0 zNOkZ7^4ey!=58-!ZVKW8v`!s?`V_ljeQmsIOlgumH0g1SPn6A3LJBp-TA6(=lIH$<9G4aIO+^LOmGfA=-Tuy+0hI7|~2b69Ywt zS>lYVoZ~i?F=?wiuoXo+LCYq1C5gTc_^uJtKbe4|BK< +

CASE STUDY:
The New York Times: From Print to the Web to Cloud Native + +

+ + + +
+ Company  New York Times     Location  New York, N.Y. +     Industry  News Media +
+ +
+
+
+
+

Challenge

+ When the company decided a few years ago to move out of its data centers, its first deployments on the public cloud were smaller, less critical applications managed on virtual machines. "We started building more and more tools, and at some point we realized that we were doing a disservice by treating Amazon as another data center," says Deep Kapadia, Executive Director, Engineering at The New York Times. Kapadia was tapped to lead a Delivery Engineering Team that would "design for the abstractions that cloud providers offer us." + +

+

Solution

+ The team decided to use Google Cloud Platform and its Kubernetes-as-a-service offering, GKE. +
+
+ +
+ +

Impact

+ Speed of delivery increased. Some of the legacy VM-based deployments took 45 minutes; with Kubernetes, that time was "just a few seconds to a couple of minutes," says Engineering Manager Brian Balser. Adds Li: "Teams that used to deploy on weekly schedules or had to coordinate schedules with the infrastructure team now deploy their updates independently, and can do it daily when necessary." Adopting Cloud Native Computing Foundation technologies allows for a more unified approach to deployment across the engineering staff, and portability for the company. +
+
+ +
+
+
+
+
+ + +
+
+
+

+
+
+
+ "I think once you get over the initial hump, things get a lot easier and actually a lot faster." — Deep Kapadia, Executive Director, Engineering at The New York Times
+
+ +
+
+ Founded in 1851 and known as the newspaper of record, The New York Times is a digital pioneer: Its first website launched in 1996, before Google even existed. After the company decided a few years ago to move out of its private data centers—including one located in the pricy real estate of Manhattan. It recently took another step into the future by going cloud native.

+ At first, the infrastructure team "managed the virtual machines in the Amazon cloud, and they deployed more critical applications in our data centers and the less critical ones on AWS as an experiment," says Deep Kapadia, Executive Director, Engineering at The New York Times. "We started building more and more tools, and at some point we realized that we were doing a disservice by treating Amazon as another data center."

+ To get the most out of the cloud, Kapadia was tapped to lead a new Delivery Engineering Team that would "design for the abstractions that cloud providers offer us." In mid-2016, they began looking at the Google Cloud Platform and its Kubernetes-as-a-service offering, GKE.

+ At the time, says team member Tony Li, a Site Reliability Engineer, "We had some internal tooling that attempted to do what Kubernetes does for containers, but for VMs. We asked why are we building and maintaining these tools ourselves?"

+ In early 2017, the first production application—the nytimes.com mobile homepage—began running on Kubernetes, serving just 1% of the traffic. Today, almost 100% of the nytimes.com site’s end-user facing applications run on GCP, with the majority on Kubernetes. + +
+
+
+
+ "We had some internal tooling that attempted to do what Kubernetes does for containers, but for VMs. We asked why are we building and maintaining these tools ourselves?" +
+
+
+
+ + The team found that the speed of delivery was immediately impacted. "Deploying Docker images versus spinning up VMs was quite a lot faster," says Engineering Manager Brian Balser. Some of the legacy VM-based deployments took 45 minutes; with Kubernetes, that time was "just a few seconds to a couple of minutes."

+ The plan is to get as much as possible, not just the website, running on Kubernetes, and beyond that, moving toward serverless deployments. For instance, The New York Times crossword app was built on Google App Engine, which has been the main platform for the company’s experimentation with serverless. "The hardest part was getting the engineers over the hurdle of how little they had to do," Chief Technology Officer Nick Rockwell recently told The CTO Advisor. "Our experience has been very, very good. We have invested a lot of work into deploying apps on container services, and I’m really excited about experimenting with deploying those on App Engine Flex and AWS Fargate and seeing how that feels, because that’s a great migration path."

+ There are some exceptions to the move to cloud native, of course. "We have the print publishing business as well," says Kapadia. "A lot of that is definitely not going down the cloud-native path because they’re using vendor software and even special machinery that prints the physical paper. But even those teams are looking at things like App Engine and Kubernetes if they can."

+ Kapadia acknowledges that there was a steep learning curve for some engineers, but "I think once you get over the initial hump, things get a lot easier and actually a lot faster." + +
+
+
+
+ "Right now, every team is running a small Kubernetes cluster, but it would be nice if we could all live in a larger ecosystem," says Kapadia. "Then we can harness the power of things like service mesh proxies that can actually do a lot of instrumentation between microservices, or service-to-service orchestration. Those are the new things that we want to experiment with as we go forward." + +
+
+
+
+ At The New York Times, they did. As teams started sharing their own best practices with each other, "We’re no longer the bottleneck for figuring out certain things," Kapadia says. "Most of the infrastructure and systems were managed by a centralized function. We’ve sort of blown that up, partly because Google and Amazon have tools that allow us to do that. We provide teams with complete ownership of their Google Cloud Platform projects, and give them a set of sensible defaults or standards. We let them know, ‘If this works for you as is, great! If not, come talk to us and we’ll figure out how to make it work for you.’"

+ As a result, "It’s really allowed teams to move at a much more rapid pace than they were able to in the past," says Kapadia. Adds Li: "The use of GKE means each team can get their own compute cluster, reducing the number of individual instances they have to care about since developers can treat the cluster as a whole. Because the ticket-based workflow was removed from requesting resources and connections, developers can just call an API to get what they want. Teams that used to deploy on weekly schedules or had to coordinate schedules with the infrastructure team now deploy their updates independently, and can do it daily when necessary."

+ Another benefit to adopting Kubernetes: allowing for a more unified approach to deployment across the engineering staff. "Before, many teams were building their own tools for deployment," says Balser. With Kubernetes—as well as the other CNCF projects The New York Times uses, including Fluentd to collect logs for all of its AWS servers, gRPC for its Publishing Pipeline, Prometheus, and Envoy—"we can benefit from the advances that each of these technologies make, instead of trying to catch up." + +
+ +
+
+ +Li calls the Cloud Native Computing Foundation’s projects "a northern star that we can all look at and follow." +
+
+ +
+ These open-source technologies have given the company more portability. "CNCF has enabled us to follow an industry standard," says Kapadia. "It allows us to think about whether we want to move away from our current service providers. Most of our applications are connected to Fluentd. If we wish to switch our logging provider from provider A to provider B we can do that. We’re running Kubernetes in GCP today, but if we want to run it in Amazon or Azure, we could potentially look into that as well."

+ Li calls the Cloud Native Computing Foundation’s projects "a northern star that we can all look at and follow." Led by that star, the team is looking ahead to a year of onboarding the remaining half of the 40 or so product engineering teams to extract even more value out of the technology. "Right now, every team is running a small Kubernetes cluster, but it would be nice if we could all live in a larger ecosystem," says Kapadia. "Then we can harness the power of things like service mesh proxies that can actually do a lot of instrumentation between microservices, or service-to-service orchestration. Those are the new things that we want to experiment with as we go forward." + +
+
diff --git a/content/ko/case-studies/newyorktimes/newyorktimes_featured.png b/content/ko/case-studies/newyorktimes/newyorktimes_featured.png new file mode 100644 index 0000000000000000000000000000000000000000..fad0927883a9310b252dc3ea75172b7bba5a9e0f GIT binary patch literal 20189 zcmeI4cT`hZ*YM+vXauDwB8ngcML?2}(EHGf^xh$a009yp3B8R7BT{4p0ckq)Cej2% z1XKnP5mBm$fT(l<5ow`)7j-n_I8UDUUGMi_vQjqZ?sCrW?0xRJcdg4seO>hf^vCEy zAkYC#4HZM+w;%X~f`0m@RuCKg+%g{hQob*ePO;LFb{$=U@j#E7e>O7NGPBICHmt@C_gBk$otvJ zUw%|DL^J{GNy2*Ixp(}c96Y>9auCQ)ps%kl-GiH^?K*gxzq4)?>* zM3SlxfZz+#A6gQP{XH>oLk!Wwn}Eis`T#@W{n6-2Sf_83^EXdBNB*_%7(eX4nC%?- zZ1yP?KF^RWFji@00tQ9$AQ*dixXXXeGX1|0aw{wEWR)09jGJEvg~sA{$O_29zc>3f z=ifYHR8S<0{7z0ng{6SEF+yA#DJd?`KFjtdsu_e25~Xk)o77 zC^WzK0o0HqiiG;Fyin>=_;Ex_J9!XrD3UxDhjPZiJ@L-6@NbP2nqMj zqLNPHjz|dv%JCCK9UV&Zf4ElhKzr|0%oo>=|3lXl=KpZ5N5BHB9m<_TXQ%i7(-1#! z^N&y(SR#<4{uH?jWbmi;%n0N4_tw7-xMM%9OP(kK5wlZua*)5u?(0JKb)DL=`Bc5q zDD=*XCXe1J3Jd`FZ>`_D_0{r=8TNlWfuH?TnE!hWUl$DiKNZwxSD&>0-3igdiR6nS zU=*E!ocM3+{j=x4n|>cc7QS;9aL4|AGCsHdeUARh+JDZe?9SpQ4desRH`J$jK*+-X z(e^F!XMOXXd3E>xq%VOK23~N)&i5Bn%ERBATKs2I%ERBAezP>eV@dMJFVVhhrm)&s zfj=(|M!;qPBQGu{CL$&a|GxQq3w^8~#@$>63)C=iCo_?Vzl^>=@sX)U;NrS3P!$lPlr2-8XB@L=F4Hs2JlnOLl zlr*TyG+a~>Q7X`IQPQ9)({NEmM5#c-MM;CIOv6PL5v2kR7bOj4XQE?7ga=*3N&1lG^olnTvQQJ zD$sCI(x58Sa8X4>sX)U;NrS5VA92xtdk7GN2cG=%1s?I+E36|3JRZo6)-coofdWo} zK);>`f!4Nx-zgBtTNnhIwg-WvuYy2FJmT$|)j%LN0ZkP}W4{;g(w+TSd&&1YQ&MIJ zhJM!(*#|v*P>Ej&4AoHQriXwHcjuVv@4^b|JX%@^V417phD0XYswfp??mKz%BrDH> zQ*CNS;%CJ%>+h~CZVRJgACB4j$JowoZw9wJEkBkbhAduNxht~I$%1Xyz3cRJ$-sk- zyFfy_?zvjSZJnHLw>~X9tG_Z+13GI%7q{fK^~x zVuUG-`}Pn%R&DWDxp$$Vh=>XQ)Vvcf^%R*OwYO*J=9iY1YUO@dT1u3!8F0s_p^c4i zDHZzVKy#8|Hsm@9EpY9^knf>Zz2{{iP5pZr-;R&>SqjnTc1?LcJRB7qxH30fj#km4 zZ+Y%inIRi;*JP#-net8|;{m?LV{2nkPR6&Jm6PS@Xof1Ws&*s)`dfN`Ma6TLxdvOh z-A3l-=7!epH*baromA-x%PJt>xfAtBOF^OWLZg9Bo?M;JUb?*!^{3|*S)1YFxBzU+ z;}Sb}sWOkykP!J!FY&Zklp}j~RdIiT*!26zNcB5@7cG)Dhd9L2I)0A7UQoc_@9G^M z8J{f5>ox-86A}`VkT5IMw|;!b*vxF@NqmjR%hkD_+Dg|dvV=tgX7Nb$5g~g=LatKC zhsRR|lP=-E_quY5xRNxzG#w1HO74@JtKmQY^ixRk=-sX9bdmfg>>CH_mS;6JG+s&G zeC%RPUY3cVqni-)s5x%t%T&UE?%2R6+v*whm+&tLHnYSS~> zbzdP`z?a#emVf$0W;1(}jmd$b!J*S1r^2{5KE8=lR8%Y)A)MMyUVLDsstUFG4Kp^L zF7N9zL|*!N&wiKg>R9U*RyMYr+}xCo@gvS1IY-AQFY^Za%+v3u-zZsht{ezhnaT)R z)^=36-P+U2|o4t1B0TwYR_5=E;-g#l=8Ral*7C`AQ+&(aD)pHvW!F z+v~ZDL+8(jrRz?t5l61c`1ZA`51y7f%5#O>k-62j$(Yae+ugFuH{%oQ-2Ok~A#}!JJg$&$E$*xXrNUERo!rfWGC%Ljo#goLyU z4g320&$d}#mk$;XURcZ7*U1pZ9r;?>I``?*r@!9&#ekJ7==l|_k^d!z0}S0$OA8AN zr48cwd3k*6UAgh`@ljDx&aIcCV`5H*_gD(~)Y)u;_dFP{&@VAnS65GFO}>(NG$CMF z2nO3)epidqV+_PRIj+c@gQIs1VvJT*^qV|%fL&C0Y%1T^eWW~KtS&eqp~|x4{>blF zdz4H2M^;vWvE7yQeld2&EM$>OiP_KK9=pi?H$yAC>F6}qi!QXcX(R-`*>!=Aj?U}& z%F@;*V)jMrt>{{xrD@K^3`OPN_*v<%P2_Ib$K-3Lk$VdA4{U99OlU36E~ge78kw4k zFQ*{9IHj)^73Jxs+;JUyghUT)PJ~6$*FJDWZW0I09+bx1vcO>r3w_dN<&%ex-P-WK^d?pOHgWw@Omqrj)}c!;{kX_=k-K08ylN$Cb=8V z^VjMJtMToCTa2Y?rAuh?uSTopB;SP+&05$q}s=JraO!l5ht)BO(W6IB~ zj=Lo9_sXqiu-<++$S$_p#?d(C*~1EVEVhtDUpKV1Su!ti!)hX1zd_bbyuBiVE}}%a zDJb|>w&AsEb0D`itITYBi+awlGZmDU!l45Gca5CSKUpbhKI`h{W&=wmNhi0PY<24$ zYE?KfIiWDNc2TIkufJC&<4N=p7(0plz7H3Zn=8~Vef$zo8|sYt9oJ>_ijDL4pD=I| z)j|)H2iGa)Uza2L;PH5e{?=pH+r;E>dqv>NJCiQtdEG|m75tRd zL5$iP^wzGUN8lAg>&aqnACFznSBtr2E+n2-Qc++6WqN1WT;zUE8LvxHp{UQ`iJ^QgcQP%${uncuum=!T+P z3ENxB7Zlcdt3z!Y1E8{yp!GQ$sO7DP^+9Xnu^eIClE)d3@-i^)GvH@6Ev{-Y zFXSJnIo=%Og}x?cV!{*t;ubc$cgBz@BqMO`!{dR<U(l^zNTYniQ}9hNir*Wd3i1D zlgcbc6~SNNdjjC3f24PhUghSmjgGxvdWZ?27s>*V^mN>C&BcpdsUe5mMAL`h+ zpjFQzt&(_xPYerBWZu!5=wP9fj=ia3addkFj|j#fW}F0~rh96Q9O6_`vpSqAMYGqJ z-(lpe*PkylutAX~CgWpc{)7E9Buz~@-?_o=&{w~N>jVAcSe!s02U_&R*+yWdg}tn* z!M>Yh=j$P^zR#_yT$ftz`iDn4>t0s`d%t^*;Nw%{OP0!jx1GXGD$xXC5GTihErN5)n=83|ZXZpIQtF(acI~YySA;=tz}J{l{13 zWTg4X%d>-d)2`wA?$*}rWv07FpO%$fOgP;Ux~Nlc>rrWOm_EX^xa8)|YY1F`qU|s+ zX{~x!pWY(5R$6vu$Q@yR1&`Xzb27IljNc(D0RJF-;%ws!h^oP{ zSf}Y*g_oZpysNzN$!?k}04o1FeK&bzWB*-Jb+qUE*WwL{ZFYyv|li<;Iq3!d)A z#y#N|JJ*lyk3CFGRRr^fsT!ONx7ThTBM6)}D(=G&CZc@Y|k4(gKy2+YTQqF@5EqgsCG2OReFf`)n{e9$R7o1uYjo2TJ@r-BKCA{nKfU6Qvc9scbCtEp;}8~`pX(N=uLijN zqD4vHbdhUn9GJl9yOT%Rww_==ViGtWr`Vqx{-A-&jRs@=CtXU+=`*9K28jck_@);`#3 zYNjb{Zx~dR;T(6fG^5qy=)-;iPYug0w3k+&B2BA+FRO*Lx+gt2sRl+tI?g~KJb!Y8 z3%7)EgTwyN=MPw#gW~qoHzL{h>$JNICb|N}AlsRLUDE5r^MyLZRejb|IP^LbJGa~_ zB%JxOeN1zUer)|=l|mVnN33_f`ukUE_r39!=4^@GR0X4=9N1f0Th&iOwp1eI-nsQZH-l zkUHJr%)g<8#Kih<+Ny&0g=sJ!WVz4I8@F z^6zED7x4D%d8X;<=@z+_mr3v%pXrxJ(b~YK#2~-B+s(}lo2s^(MTwQZR@6K- zZ=cfDJ2E~o!Y#*JdBd;JN03Y~`-v+&^2=rJ-SLNb{E46_}LEvC8Pl@iJ{)oPv zU0TXzfkM7S1xI8`_tXbJS;4A2MtU8r9(y|KZrx;Z< zSXJW&byOY@(@9OyVcbgzZ~U3s`}!<%wnRjMoux_QlotMGjXPur-O|+stjlI;+OAc1 zG8=|R%M`TOij0fmf?QhFb!WV~MIJbkKvZKB)I zNY&u3|Kj-U3H;fW8|`EC3=GQPX79DE80e1Q#aUiK~{H{N-HQ)YR(*D6Z8Wbl?he#5V>W8uRM3#%G;@4BH?SjZlQB+Nu3 zqTk##CMK%OrN^o!$V+rKvVH6r(s8o<<00r=!>tdCm@NJa-J>Xr3q!CrMT09>PJ(4S zA3B0FpJd2Xx%WARBU!r+W+;ReNZd4 z+#K!e$l2chag7>v6~~-X+wI|Lv6Q1n!I@DX2-(jj%LYY6MBGh_2drm3CGo?z1;9!v zz zAWgnUg>Fw~J5`if3>RJ&+^P>)*&1E9Iy{!B8bP;r<%DEd6H5(aUQxjwD40$NOqRnfHbMvdy@#u)j$o z#lazt3%C?N)qE>b(O**^cGYUV+6bQAA`6S=;wcYAtzSP!!S`>bcIhuK+K zSy-Np&RGhfkSRa?{IfumARHd*`sYK?`O&A(c!yQl59CbL|Vsi5+~!{&V>NG;O@N!iHV8jzI7doK_Rxbj>fDy-KS2SDqrDv z!iyg5tSj5w&?*@9`VyziuvHZq82#nTm$$2{y5hUK;$Mest}FamSI%^Qcqs6StBICY zRF6r`AnvBPnW?Gg@%fqJ`RC=ld0V&5`cw_v?~J(i<+tdZWs4gg+`po!a6y$ zSF0W(A{*P47Zz|$gM0SSZJ`cbPfJMfS$#it!(B-OtEq`mRRvb-4yOc%7UH3n(%|{} zyoely1+A@X3sX#9n@q70aiaaDHXO3{+^Wx5BTnrDR(#!jvqBR9cCTwaX^k-WzS=YM z-h*+Q-Eo{!A37v12=mBGXys*E1yX$n)ev8tPS#4^XdC}Yl9BN5GrxwzyS!) z{w5dEWAlb42DWpATHb8PDEZfjd)Wsz%lI`^^`Ot5>u@jTD_OEEOKO-#r@2!TL|wYAiZ!RtKujlstPf3d8Kx4{d2 zpoV3j3Boln#KGSMqT-BjbYau>cDV0i?Bd`Y=J(n~0RqAHcQ>^RwA9y=b3%9vIs9!S z6zc5@dP5)z%AvjvPM$7-Y>qDX-F*~cyDtV{Z0^pAFbhe25q)1Z7dLmUaDNw*a064P za8D;$XPB}Qn?k4@7{J>l(19(~+sh|FE>scruW{wT`@dfc!`S}SB+ye4_U}bm>Kn4D zA^cs~B!#2|okS%>*ra8JM5RO|Bt-?-#6(1;g+(NVMWh5pMCC+ex`uI`|5U3W*4N zd;i_nzghQAqhHi*JM1UK@mrYIW@2yJ;NwV?iJ2<)f{Qa8u z?^XSK=jty0?!hk3n*IoHwttFJLw+q9*`eIGONS5Z@4qWk4BRwyc6N4k zt&NXYYcg+9I$gnLWGr?ZP1TN=YcQ$P$>Wm``Qi9FFqod<`Z%-pNaC2&CVq&K%Qwa;Eo6E!Qb9FLseYF&^>%h*=Ua1m~J40}%M5Z0F z+`rIqzlqM+wxeAt=!YpODJenXdSVqC&81)Dj6f`-HHczjt}z(Q+egCR*VlU^&kk6x zPw{r7ygx~74eA$51#QjTWU;xK^`j``X+c3jjqamAb>q~6=P%~gpICNoA_lI4w`OWy z=Wu^T?5~Z!&Er2=f1Lw1I$irBWcOeavU$4y^<$e|rb}orHJpzuLlP&f6QcFnPx~S4 z%~32voYp=W5fRaO#NmcMnS+N1--XY=nmui97O--gYo?cYv-7!&{Ig>CZ5&5 zKKW>*GKwy)?IB)c?HKAh1B+ED5|v0V?y7$K_QmO`*ZgC4MMdV-FTcLLP(E9y)UMzm zE(;${Ah4`As!FP>t4pP$Mujy}@fcToer&ZQ`?e#$+iTL{I#H`%?6cTSzHet^!?Y-! z&~dOg*J3@wjGgxTG6S(PhLujrsz31)JXft$lg>sNzL&pYJT(W%d^A1(Wf3CRNdy= z?34$I6fAoDT}WWG!uJ+mgdid!E(odlB<8H@3`%p?Z?9V8Y?;yoEw;znXCg~bx*%jogiz=l%m30hhHWnBuo@bH<3x;sKcNg z?VX*S$2*^$Dg(D>AoPn8-5Go}h3m~NE!Wo>RDy(fS5}gOR>sE0BT2-s2N*|+-jyq# z92|JOubrT^i>&n8HA1>QQ^Jofq;Z`n`q1w5@_Eot)rwp}Yx8uhp$BV!#;7zBvKhPs zSaD=lc@-2c58o+&XZH8^&%N!v@MLs&7)wtt1}w4-{n)VE6_VIU;Xui4aH|oUTbr2A zUZ{er$?l&Y{lwPxecI*oMQocVcz4OR-Enkt z(=AmgEE$-1L_|cF*8&ABWsX%bJaBJi=;+6X&)@>!@IkOM%vHqi%}Ow~?cx5{mpvNY zCUpkeDE1O?H&K4?4!*sO=Kuyr{5pMLYisM-vuAoaa>V2;`#&FY-1hSF^7VIJ$W;DQ zppr=m=2g6PE8&l;t1DV-w!tJBf$OgXgQ?y``@lpqUw-Zh;C{rvrZAn68lifuS{vIG zs+eCm^QwEnim!DFzPE9XBk9X!i$74I33HVrh`m@D!lp(&Db(jT*U=#qP9tXwBA`6E z1xv`*xb?BcS{mO$K-WG`jy&G{$XjerYzr|_B&kw=N!!jbtONS-M(9x+axCkx7ITQB zBMF@MMmr5_bf`B;PhGw(RLe2%3w{}hv0GNuqFoz6sk=J>L&%yjH#g@ao0wB4v+4Fl zMtOqBz)9ldLOja=&+E@&L^U3H^O-jnq@@w1CKneM^O=t`p{V&x$*xuVX0T3zI2lFk z2S3`yOeloxe3Cq_1MWvaK+q|0lrUipQSBuF9QPCs64|a7gNpUN|(Z* z=;`S#Hy1j%3&ePMP(cL8I2>Xdy}iBQKn2bBFEf)~QG|1Hak(yb`x(prHO#pt3*l+y zdp-V%TI4dD%LB@6bQBcEFM_>m#A~ir+h_$WMSNEVKVnUv61J_Z*kg*iMX7M9V-jzo zrUus4?pk&~e~ycbJ6CVy^uAVK2Y+VrLz6|!_tFq}Y6gSka1pTe8LoyT4V>~D(t_US z2XBS>%^Is=ZBwHtumzgR!}ZscckbNLa*9fZt92rlSx7qXcDRP@E-{O{LJ<_nhuUzA zd&aF-v9XZSdd28V8Pd1LNVlZp_ji=9R*`8m`giVhJ$WJ*^dnb^`-R_E%`6zPb(w4c z7a}`pA&r^Fokxi=gfA!fz1z=uW+o;K!!B@@orR7%uZ1Uz78Wyuw!>h6>qJYBBR%y9gLX{WLi``lvt;uu5(88U;# z$w>pW**#ZG6?Ced`%YQ2Ngc@tP8B8WlL~L(D=rp`!%m5?Z}>_vV5~dkN~KLz0)m1K zUouM$(&@#eyyh#;zWN%2@W06da$r0jscP(D`~LnuF{9M@>(?o1Y1kVO&qnq-@(bu3 zNIX{7*7CBl4N(I`iM~~{Fqq-Ji&b{7oSYnj#8`vi;A2wdOGqhuDk&TE0+Lu{@igA+ zk+2=t{8OJVe?oLlnwRFt6@mC%q7Q0mx z?-SgCQ&ep2?kaoE30TP@mU`#sEm^*X9B<#<_d?-S=Qt7DC|l+W?QCwMvMO{R3Gv|& zJ@ll>z}0KH+mXchXkpMJ>iqZq-d?>$TNzSZ;9f5sezAmmcDMf;CI!TqTW!QRU-{%} z_7A_mv+i`8uGME`sIRbEy$M?wNhYlnXQLuvRVpbgTxhkG=P_3g!Z?SCXtAOwu+Zh@ z%bU|x#2A7`VRK}GFs^@#Srd(*MT^zZr>CBLv$J&5v$$8t*v$Nv=H|&#`8u@WO|&-g z4OWGSL_+G;dwr?PyZslx)<)B^Bmd@xGaqej?I|8X>t_k4-b=k<4nDL-$oirmMTgG9 zcKzw;>6hOg+11P2`|No-&#UnUld4&5x`EWwJCAT zaD)4!E{;K>#gGWMx3>c;kCw-y=j7(TQz$k%GU5aPkTLPv*O!?ZXW2zX3^X*yuDdkq zD=RCH!wxsP>ViH8ww{5A$1E3^WgfjhN(&H@@voA+JhKj0{YX_{ln4YO!48-Rq%|Xh z*tuc8>s!X>9PT@099&#n2SsBUjI1)gv5dBnQBkF(rM!poZ;W3)U~U(=KUuQ%zOML> z&%O?~UcqDc_gn%eJD)?|<_jWH#-z!~R47hGW@{45OA4HJm->vKnRoAcd}sveU8}Gq zd?@awan4w=BvnD9S<{R>)$X(KqaU-~P$FXDow*jq*+4_XDHJ)!={!gR#?L7*eBPfr zUnBhAynZbUU|Q@A?9;T9?S+7g;~fM6nf`iFo9zo+Dhln(KX2S}($dmo%2?k92M3>L zw*>sj>cVJdvSf%jfkOo;V(m+9O^wk{aU=!l&G{6bS-0ez=s#!Q@km)D^zj}8%Ud-Y zc=~Bp9iEOt!7tUz>zrR_UvsX*H1;0m3tErysGp*RDL;Z`RsX3&%JWu8g$}Z-CGjMe4z3;+H8$WI)q3XXz%Tw(|=o?n?E{#?gO?Kba zH7OyX(`!M@!UD7U;K2isG$#h=>Os(oyzbT}K#l(d=^4G!4HlPspebQ8$zW>ZnaF@b zyEr?Km&((USV0@jn=O$D-nd$~X*Ag+errvq?bv*w_Oh%hA4lB_Y04G8so`M4kXSw~x2W!~V+FO7@sj(53M7k1a@7C!Gt;g6w@a6f)S z+)Vg03$sBqk6*j5MOq-d&8ws#>V`M9+ChlR7MAy4*8+Hm1z(Nn8B|wKZ@o7OUhPoo zX3u|rvI9L1}y6wlR0c1=o>v#@R6+n*3a$d{0s9;ww@!mmb0 zIlEeaEcHcPogY!+h5J8~vB9tCY;Pa>JR7!+NujKyc8g4L1|HIA({lhUKcmhe%F~yz zMknu0F-L#IiI5M)&^NS38kC7>>UwX4w8grM=Im%F9xqh<;$-*Tn>U2``12k2NxSd* zPptisQC2+zh{$WHmp*xMAcl1fH+)%r=`yhaI?C`S1h~NQ@o}hpXSN4TSV@iM=W|T5 zLd+kgXq#n%)eZFu_|#OmK{{{$(6_gbbd%Ev;=2&bLO$pFFImHKUc8VOO2{+(-OU;$ zjoqxy&SK9Yh^Dw@zqS`TY$Q2u zhzbZmEmw7Q?W_pYMdS$SoF|f5qj|n)e;v#4OYUTZ;aw^L7IJv87vx7hsWkeN0E71U z1<=MoJD_YlvJtPc*tx#)QuerlASHT-D^nRUjn)9bKm!jc%}@D9K~kbQzWeoBz<>ge$N{H$g2 zA|*XN;kKGcBU8rc2WhtAk`gAFpM!&N>@t$f*cq%o(sFzqB2v3X9m(RREKPKSB zdFQ`5#lAJOV&CkHxi~wUKD%L;Lhi2UhQ-ds5&%+CuvM{qh>*-dBEbX|+%&j?l#q}j zWM#GQvK4TnHzFTsG0oOpB>X)*IkIwcseZ*c>gSyzuENhtGBY#j!c&V~#|tWVdt`R% z4Csg8>1k0F8{-HN=7@@Bm~~y8+v;DYRBxOQq&V*@E0;*6yg$ zFh=BwQPA!b0U@WCi|~2PE91+wD_*T(EV2}y)DBWCV~LlP*biTviS%yzqmyC-xN)Yq zu%Eg$!;^^_lP(_lWD<8JJh|`Ul1Mu4BhhsvlZ}&4F`v?5 z+bmZlIF8JUFu)YIy{emoID*L=I3*;gc1Pja=Cw3DEIf1OEmlK`e#ZJ4&xAAaNyzxH zjn+h__mPGfCFBRii2h2kqzaZ3VU_&rHS&s_6&l|hN-n*MqaJU=hegiS^5dU-XL#6W zev6QhkRTejMgyro0c$N+2osZh!CSn9q&rq(;(h-7IRjnwFfZ=DDu6sU+v*frPOV0% zh;T2RUT1S=TxJ0vv;F0Tw3&6B<54N)BH&{iR}Upc?E9@BfX=&`}}fcJ_MkRp&{ zL!CZftd^B#Y<$l9v%0NYm6TKG@K_@uxN&Na!tVx{$sUXPuhCF4e#xt=<7QIqAqo#Z z6?*>;d4d~@&s!&+Cf6_kU`OiG-EypoQ|8IhK`8m&tYeaWG1S;aI7uFw*H{ce#!H{DHj#AUhjSOnN%>w4u7XdT-A+N>S$-h|byQSw3L` zw$l5tRn-#8Ua9KLPLAQRT+8c(#`x(au71t}dM#wIhKg#wMwfP< z;^$mTT4tuS8XH!yAzxP4R2|6DCbfDu(v_}$n|v#IWeAz_8BccdCVl%U#Ml zs0QxevdO*Fk=LSw8CUDm3fYL@rr{P?Q|%^6HiX({z=->PD-oLpt5!0`TeLsI&5QAq zPWW=x(ym900r>K+we`itMOxLjeXq*ON`Q^5pLq=(fSlgVj^h~!yW=%J`gYjUKRB3X zd8KXO520K1@xuMZbzvCPV!7LoqtB)FPsKntEQ4{@ttqU~i0u08?2KN`9oJcguE=)4 z-=^Tzy=wPvW>T!+)!{@rY+PQpN*`CIcY6@k3D%$AYnnzl-t&)ksJR22d^fTWWcsVO zc7M>*=4HC2R78Aa#gd-54oxMnLU88}%<{p_4U*As7hOk_7kdI;j*TTfvGhzRa{`Wh zufvs?4(l=QpZc&^6}?hid`{tey@RXw?%mTN1ZW(ITjv*I6F^T!Mn+3pCtMQMVn`j} z5)hU8BrfGGv>H3>_TkCFdll`=qqdk)ESDd1Zg_BHwXKcKDRhr7!6IDmGWuet3*g!a zAWuACmJemoTO0#mVYGstvjfP2^EpKwwzjsMe2MXzk5M5msr0D_FNpn?`=i=|?g6Nr z^I&ah8gP_q$qGFyGI`Fumt)C85w!2J^7Dg1YPB~F>jB_*`ZD5;e)s44dLA4-z_BJj zIIUS{k5il_NnIV5U-w2|$qDkyuVN9;5;8F}>nq;M4UzO+fm5+3NW3xd1Gp04VxWRn z8)w}ZLcMaAu3)OTJpG!P$G!a2leRpgsc>@?J@11Ep& z{?tEbR*_%`{p5OMy6Uz{j^!X;ek&avoz?|rY>ZFaYP-<0mzUqIKAvNHWhj0zM?>c% zUbtoZCm~H;!^iohs1w_$tNVa6x~t64^!N8057<5@du4pf$wpU>om&-|CgOru5{U;h zM3aZ#g1?Jac$I z6zlmAcOfCz4m5=D(9lpzOS)HdWz6tGX#~q$MZIAhd6d^J#TPj_ls)9V{Cq}jk}NZs zmJ5SEv%l;HqN7S>uD4r0jk}-6EvkVuFjStY%(xQ`a%R)MlpR@JX zCaKg6bp`*J)h<`fh4!;Zi$q_Bx+KKL-FUDK^wEBO78bI0+I3}&(n~1X0PhL=&6NW- zBzfRLEuUHApbIIT$lzdMe0)3*1)zG2^%AFUQ{O$s$z!XHyf8o5r+vGogt*Fj#Fmlx zW?Gc_@HZ)n6yswR%_l17~v8p4c zsgH50b$d}RRBwS~3BOsq(8L4zAiY!ad9zsrV{=cNUM^c1WI!aQpDar3`XL| z-$NQ|KnYJ$(fGT|0I33a5T}mR07!~DrZ!*#y?Pj{Y}MuCEky~3GRx)SiNHxe)Ei;{ z;Fgp$voldoq2Lq{5P)ZymPUx{yQ--u?mZB34hy8LUr*}aj3sw z0-vgxuy9)<5iMU1`?m=ofR4JK{FZde9^#eun$N=R5%cUfYq(9Mn_h4yc>UGGii!%L z8o-I|J}oUREiBLz$uUx|(g4+#l7;2h?(RFmkJlXMb$W$s-@n%_n)ZbUEhYiOI4TyX zdOB01Tdo|lG&F>KE??nQAZAfVh>2`(Zf-79j5v^ymGa-_hy56I-+Yw>O3_ffzj-5{>g{-g`8a+St*X8g=vCpf?Di z5)`Jg*APaG02zT*`1REZlJwOF0!pr_ib|lS@69`#FwnIFUYfxvPjC?BTgQO%3`W&4P6H8U2&hLaGtgY zZW(8kO^zgpyHom(9kx!B^o1Y6sA_9#DR5Swj_&iHIRgsJcQCFl3HYC}BXLG>MTEWm z%BzPQO=eBCnarF8#`-KNdAYe%nwmLW`WU9WjSa15lE(nRgY>&R5OWRc3<|4*2rFF! zgL^<(2mCjeK2trw&u;Cw|9&(3#8+#fd&-@*!80Q%x<1Fajc zcjo>3tIJEwIm4UPFpxSx>{M4*|Gm4+%AlbanwOV%v;08?Ws#hiSoZDY`dX}M%c@_y zG&D5Hq^w}ae*iC&Elkt=%)9Yow;!Z5`c(>)scSyhi|DKKTwas7xVWV1s$6X&h2GYVk; z3JMD8+7SofSOJBaERoH*IACL=VOAY_68FU+3Zx2&uUwA(O%`q7q-oaTYqJcWJ>7g8 z6@L`_Br}uLQJt5P-(;&d?69}5k5S5tj+t3-gG<481;TlBM_ZdbUYBX@#}5V;mj2Lv z$I~cmHFpgk>w)O20#PT4PyJL9-L}hRhE_mJ#=ZxHny#4ZXi$yjxdOILyBPiQri><_ zQtaFl@bq~gk~OleF9f~ne!yV8|Fn9{j_t3&TeA2y=Z?~kk=3`PUgSzMp0;i=xRfCNGY?`XNun9|ool{L?UclgX1SGNL5d;mGXTHzT; zeDYYuzwc?sPGO5Di;OSZ?NA_4vC%e(sr=y0 znHr+XEIx+SL?pJl6QFtQRQ~2kAa}B>R|#z2mKrlOsZ`>YZOuNBrA zQxDU*zmi3XqI%rl*Fxkh?TXAys+iks;YO#d5k?{z{wY~NT2B4Q{|!H*&#udd`6}@O zSL`ttKmSiaA5lbM2dtoQEM09yF(slQBq&h%beXpZw= z2_+E5;Jrpwl9in;T7aRa#%!%INVvgA9DS(_?)8T51LfgR0eW+m(BlS0-Sd@v#nPlK zl2;W==BHjOc_2O;yFLc4H)6!15NhaUGwsHB?n@;V^lP!lrpG@eCB>}CoSHK1u9cPJ zeRaTi9t;eKWk~6YLSM$wA~i%i%4EJW(63ZH{WoUmx38cLhi$@qTjH zhwk>BkeN{YvS_v(Dlj+eXmBCS$&er@o2f#&1N};Sxf{$tHh?A?(dLQjLUBJVT)}ad zf&?edo<_F)Wq z^etQH6hZ6J{h(%3WC4+`zW%ob>IS(_gOdr+1as8b$^aUT&ZY0=y^F^I=Uu7h(=5Yf z@G%<4#CrNYt;rqGbYJr}RPt@qSd#}8T#{7c9X6d3Ae)P!$n7CjsHmWgTw`RxK$0-~ zq$Ib#9J=jISMbg!yO^t|lTjE<27~RE#MiYo%n;9_8lD2ThxH=$Ay0_x`->kp&Bwxc zZZME(n`BIuD=RB0m0bv=*7x~+#dV=ljZI5NuiB#&0Wm$AnE}s24kDq52|u9+A19Xt;+odddom8~tz}3am;XjjRr>^t2Yd&i+(tkloNdJkSo> zin+d6&SX_4NhT3EKq@IIb>A7Te}k$O!9+SbCm~3?nl7KOG=f}WdH3!<7rDb}=OWwj zCxU=#KEFu6LC3{*!94abO4Dj`OB zfh>RDc_JYN4dtj-C}mW#Qo6Ify;lHqN7#w}$0&~Xo75g3wl_C<%x5(; zHRGsOSB6JM2F^AZPX#~>aW4~TXU1A$k}G;`d@!m5Rmn+h%|F5WHCps`d)v!MlkQr^ zqU~NH1hDTMBm?B}0SLB5lAih5*)YE9UYGYK$kXqZ)JIUreYAQyKSuk)O)@7 z#GbSD4tgaOSkqLvAIF>FmI)^|T|TAA-uTmb`zN40V3KP2=0|Hs_MNSj0<5j3Gxo*gR7=(oH-YE@rj!$`HbpQswMYqnwvl<6!I!9ueQU$YiqF^I?^3XdI3xP_8Ov8cJD~(vhSqD8R33UVVXI{@UJ# zdXRsF++S&I>p#~i2`6SDBj&Z%W$rd94!E~eqpm)({>pL|o7~;mAtrI%Hg7cjtJ(>f znNq8ZCwB%4C=fh!jKOb>^aH--@1M=5l#3gjkXa_>U1VU8=vFG}NrRSlxi)FO+^ov< z^7hWVEok|;0@R4uWU)VL5D*X(6XWAgNuMcH;0BF4wSqblk +

CASE STUDY:
Finding Millions in Potential Savings in a Tough Retail Climate + + +

+ + + +
+ Company  Nordstrom     Location  Seattle, Washington     Industry  Retail +
+ +
+
+
+
+

Challenge

+ Nordstrom wanted to increase the efficiency and speed of its technology operations, which includes the Nordstrom.com e-commerce site. At the same time, Nordstrom Technology was looking for ways to tighten its technology operational costs. + +
+

Solution

+ After embracing a DevOps transformation and launching a continuous integration/continuous deployment (CI/CD) project four years ago, the company reduced its deployment time from three months to 30 minutes. But they wanted to go even faster across environments, so they began their cloud native journey, adopting Docker containers orchestrated with Kubernetes. + +
+ + +
+ +
+ +

Impact

+ Nordstrom Technology developers using Kubernetes now deploy faster and can "just focus on writing applications," says Dhawal Patel, a senior engineer on the team building a Kubernetes enterprise platform for Nordstrom. Furthermore, the team has increased Ops efficiency, improving CPU utilization from 5x to 12x depending on the workload. "We run thousands of virtual machines (VMs), but aren’t effectively using all those resources," says Patel. "With Kubernetes, without even trying to make our cluster efficient, we are currently at a 10x increase." + +
+
+
+
+
+ "We are always looking for ways to optimize and provide more value through technology. With Kubernetes we are showcasing two types of efficiency that we can bring: Dev efficiency and Ops efficiency. It’s a win-win." +

-— Dhawal Patel, senior engineer at Nordstrom +
+
+
+
+ When Dhawal Patel joined Nordstrom five years ago as an application developer for the retailer’s website, he realized there was an opportunity to help speed up development cycles. +

+ In those early DevOps days, Nordstrom Technology still followed a traditional model of silo teams and functions. "As a developer, I was spending more time fixing environments than writing code and adding value to business," Patel says. "I was passionate about that—so I was given the opportunity to help fix it." +

+ The company was eager to move faster, too, and in 2013 launched the first continuous integration/continuous deployment (CI/CD) project. That project was the first step in Nordstrom’s cloud native journey. +

+ Dev and Ops team members built a CI/CD pipeline, working with the company’s servers on premise. The team chose Chef, and wrote cookbooks that automated virtual IP creation, servers, and load balancing. "After we completed the project, deployment went from three months to 30 minutes," says Patel. "We still had multiple environments—dev, test, staging, then production—so with each environment running the Chef cookbooks, it took 30 minutes. It was a huge achievement at that point." +

But new environments still took too long to turn up, so the next step was working in the cloud. Today, Nordstrom Technology has built an enterprise platform that allows the company’s 1,500 developers to deploy applications running as Docker containers in the cloud, orchestrated with Kubernetes. + +
+
+
+
+ "We made a bet that Kubernetes was going to take off, informed by early indicators of community support and project velocity, so we rebuilt our system with Kubernetes at the core," +
+
+
+
+ +"The cloud provided faster access to resources, because it took weeks for us to get a virtual machine (VM) on premises," says Patel. "But now we can do the same thing in only five minutes." +

+Nordstrom’s first foray into scheduling containers on a cluster was a homegrown system based on CoreOS fleet. They began doing a few proofs of concept projects with that system until Kubernetes 1.0 was released when they made the switch. "We made a bet that Kubernetes was going to take off, informed by early indicators of community support and project velocity, so we rebuilt our system with Kubernetes at the core," says Marius Grigoriu, Sr. Manager of the Kubernetes team at Nordstrom. +While Kubernetes is often thought as a platform for microservices, the first application to launch on Kubernetes in a critical production role at Nordstrom was Jira. "It was not the ideal microservice we were hoping to get as our first application," Patel admits, "but the team that was working on it was really passionate about Docker and Kubernetes, and they wanted to try it out. They had their application running on premises, and wanted to move it to Kubernetes." +

+The benefits were immediate for the teams that came on board. "Teams running on our Kubernetes cluster loved the fact that they had fewer issues to worry about. They didn’t need to manage infrastructure or operating systems," says Grigoriu. "Early adopters loved the declarative nature of Kubernetes. They loved the reduced surface area they had to deal with." + +
+
+
+
+ "Teams running on our Kubernetes cluster loved the fact that they had fewer issues to worry about. They didn’t need to manage infrastructure or operating systems," says Grigoriu. "Early adopters loved the declarative nature of Kubernetes. They loved the reduced surface area they had to deal with." +
+
+ +
+
+ To support these early adopters, Patel’s team began growing the cluster and building production-grade services. "We integrated with Prometheus for monitoring, with a Grafana front end; we used Fluentd to push logs to Elasticsearch, so that gives us log aggregation," says Patel. The team also added dozens of open-source components, including CNCF projects and has made contributions to Kubernetes, Terraform, and kube2iam. +

+There are now more than 60 development teams running Kubernetes in Nordstrom Technology, and as success stories have popped up, more teams have gotten on board. "Our initial customer base, the ones who were willing to try this out, are now going and evangelizing to the next set of users," says Patel. "One early adopter had Docker containers and he was not sure how to run it in production. We sat with him and within 15 minutes we deployed it in production. He thought it was amazing, and more people in his org started coming in." +

+For Nordstrom Technology, going cloud-native has vastly improved development and operational efficiency. The developers using Kubernetes now deploy faster and can focus on building value in their applications. One such team started with a 25-minute merge to deploy by launching virtual machines in the cloud. Switching to Kubernetes was a 5x speedup in their process, improving their merge to deploy time to 5 minutes. +
+ +
+
+ "With Kubernetes, without even trying to make our cluster efficient, we are currently at 40 percent CPU utilization—a 10x increase. we are running 2600+ customer pods that would have been 2600+ VMs if they had gone directly to the cloud. We are running them on 40 VMs now, so that’s a huge reduction in operational overhead." +
+
+ +
+ Speed is great, and easily demonstrated, but perhaps the bigger impact lies in the operational efficiency. "We run thousands of VMs on AWS, and their overall average CPU utilization is about four percent," says Patel. "With Kubernetes, without even trying to make our cluster efficient, we are currently at 40 percent CPU utilization—a 10x increase. We are running 2600+ customer pods that would have been 2600+ VMs if they had gone directly to the cloud. We are running them on 40 VMs now, so that’s a huge reduction in operational overhead." +

+ Nordstrom Technology is also exploring running Kubernetes on bare metal on premises. "If we can build an on-premises Kubernetes cluster," says Patel, "we could bring the power of cloud to provision resources fast on-premises. Then for the developer, their interface is Kubernetes; they might not even realize or care that their services are now deployed on premises because they’re only working with Kubernetes." + For that reason, Patel is eagerly following Kubernetes’ development of multi-cluster capabilities. "With cluster federation, we can have our on-premise as the primary cluster and the cloud as a secondary burstable cluster," he says. "So, when there is an anniversary sale or Black Friday sale, and we need more containers - we can go to the cloud." +

+ That kind of possibility—as well as the impact that Grigoriu and Patel’s team has already delivered using Kubernetes—is what led Nordstrom on its cloud native journey in the first place. "The way the retail environment is today, we are trying to build responsiveness and flexibility where we can," says Grigoriu. "Kubernetes makes it easy to: bring efficiency to both the Dev and Ops side of the equation. It’s a win-win." + +
+
diff --git a/content/ko/case-studies/nordstrom/nordstrom_featured_logo.png b/content/ko/case-studies/nordstrom/nordstrom_featured_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..a557ffa82f12e9172639a30014f60c19cee74e22 GIT binary patch literal 7352 zcmcI}cT`hbx35?!(ouQ~B`87&A(YSrLhnVS2pAv~Aqga*O7B&A@5M$H5LBv2l`0An zL0UvXDNYM$Ix2#0!}TTf zJ=D-Hu3G+hw5h*=nS;NZgPbEsS&2`<7XeIwK@(AYz8H5b0pY6%`fFYUK%d@*fcX9z zLUdCE{YxkdeIq_K93IUF7nc@ufJ#a5$;gRAr6r``&>MV`5>Oe41RNqEEhYg)NXQ~2 zp?rUTK)`BvM<;{{QseKnfGb6i3z6u7fIxhFe8hcV;yAoB1S%&d2a%A3NJ@$UBg6=P zSR%?-3`-FBX9gsi;DC4aAiCnPe5W&_>~UU1MGzqAzqWw!(AWR>#8|@LiUL#y@kMz+ zpyCn`4CZuSe~l&(P0;^GjQ=>AVCLt6hM1rUI4`^da2`$q|1bl(`|lSz9SBH+FvPn8 zhk|lP;vBp%Xe?11sR#m|h&#GEBBUG~WC8V|#3ZDpCB>xV9Hqoi60(wj0wpBTlJ@pc zd$im?^ZX}%6&W=Rq@;wpl$@#*6sj&O3splRH6+yKG&JBaX<4;@c(t(vA`0t({%37h zVC}zok^e0(LJf~b5pj4k9M1h8D=>1w5pe_;oClwp+Ucpo#o>G+`X~oi?CI@mr$_a# zog>kBS8ueV1|Emu`%Bjd*Z&{~OimUFmsCZ|_e@pFN{{W3m1Lv>Zwwl{k$yuo!G zeax@2Y4L9HUi05sHhzJ%spWiyCOt#-J>WiAsqJvjx^7-YsJqlW~27vnXKe zt$O^`D-lLj7SNH6dOG2uUuF1q)(1E_KRjcnNiA4H67u=y5Nqv}; zk^)5!uuOj2*!VV__kq*7(C<-jk9o}9arEz z<$9TvC^O5(%ZpnNT!>^9*`D#4%MfAIntFko6Hn6D*4~6zJ~eqLGZ*lo(p>py$Mgku zMsK0wR^|J&+csXefJb_mBEN6!C!0S$Jf;`Iz505!NHU7-lH3)kQ6g1JgiA=eN8ttGQ0n4 z)w164?NAQP&!N)1S|fpNtRpyhT9-X)8rNYXxL6X+bKSbBD&MZo{z7x`-qu$a$noMK zu($AHo%knAv?V&vq!kryY<1|<#ciyvpCgmWveNFP%%R^ft zYSrZKT6v}Q@f+(V_XVGhpra=XJ*;*&{Id{u+kN2`KJ%|bfqWwM&aD+sAjp|)0k$A> zxv7&xfp}%x<_|Uzun!+1uw5=n-BCg(n=L&A?AS`SnfTqLb{Yzmd@sVYbZaqBm$EG@ z>UIBs#T$tgp*s1R0VXa4v0s)cN%2?q+K&*5bFxVf32nat-mQCs$^2?B89O(wZ@fN7 zZ#4fX4~uVW#?2`~_er$5PHhpC6HiT}by;nF`O|m(j{%PmPi)>Ek+-4YJ6XynJV?)h zyJb!ite(5@qnG2}xOJ;u5Cv6H%EUSDERP-Qj9qc)P3X*1d04_&$7kE)0?WOqNu9O+ zYN>ELRb+Dek7K{MIgfs*{PON%9Yd2Um?Y~fq~IR7B{Qld2Kmn{>6t3vb?>Y+sCT!cG-8; z5f@DVY>z5ynvI$oDqeA2=mU%vgb|o!T4UlC&NK^IWj8v7`+P0};q{0%Psz@Zi?Kn9;=r)`SINF^g zSLm(*wmBLiLtXcBQp7JX=h_g<6c&EGWehvcmU3m{;rW6JgnUa95TB&VS$$$r zSE^sl#Ob*@Q90?iRKU7O1!zIW79L0YqR-oP``a)V6)zCT29jRP1x@=M3_?1F7#}aE z>WZsCEWii@Hb!;jk_k4>dqj6XQ>x zJ~eIQFCN%-1O>Koka-dchs6o4YfaO#=C!snB4cT2^o1<@&aexIY;yQ=-$W%+iw`6` zP9d@&uQ6<%<0v9>cyMq#^%Ca{6}H*u=g80FqXSJT1M~5UTk&52y835NXOC20Pmc;s zz8KGgr=SY4)Da@aASg(w))$Dnd-^OwQGoNvNl=q75?jh zTWwor$ZIK0ErHaKco1Ra<4AfT6b z*aYo;J-ilw512*yRSYFhz(!|&e*UEG{X|Ak%r8kO8C#~jObHu8g~~+H;tM!6^_wsD z?EaXzf4JEvlo`jdtQzZEJ|!)rNL)4=7xO=>@#Y}(+$C|F=AE|Rv)j;G)FzM)JJoM5 zoi})XZ(=|WhJzKO_!m4TDsDxloV{Ep8?yzjx2d@!Cnav?HvWXsnay&g^;QAGi@|r5 z^9y-3pV=Xo)@dkPMfV*8x4d^c%sdCSatVHKTtq}9866T5Le_l1#V>Kg^icvqhum?( z%#9!>KM)tB>2xj%?1d^R&AR1;j)8`dqJvgwCnsU)5{Wc>v)PYa9Q}S%SO=7zi#ksR z6=PjRr=6fu^KkKpE=fx#g$Qe0+k~E{kqm#So@*$HWRauZ7pi0#fE0aqsXqB++|-a= z^lq2eq|FjX`rb|IL5>%uJ+^;}UTe=E_VEUj-puVvpk-?|MROzJ?nYOw*bU=R>5>g&6jgS+afjwcGE1P zGGwyO{gdQ~)`qiJkb!NYNE052*oQ8YH3chE9N+SEWv5CHyTT~xTJ%swyP}-P_`3~5 zXWhzQ%lNO>_5D%dBR9KtT><%0T(}_{MbFE_Y-PiuY7uI$23z>GyF4baN}zld5&W{D z{NYF4QV?O;Y3VUcpT{n$-i92+BW_qXt;SsuH8C-% zqH!_6T%xJt*vXx=Y7k9c1+tExI3gjof0`-zJGOqLl=rDZB zw3xWn7ta|@|9hj0{($o;N93iXS3ul&wwvk#hZ$n|S~A}iPrty`+I6(+GiuEbwe9Wg zfv++4RKf@p2I&yHps`YQA>B+Q`hC#S4dqp24*v=btGM7z%T`}cQl!~!F+7l*-=={N z42LcJ3`x_CZnwj#yQ1ltxk$eVMwR2^<2f*Ad+}F)d~c#cNLV;7DeR#|d%&xqe85GPA7&H$qB4KrK^`Pv{pBfVNIO0q=@cMj}BRlIVf- zOFX`3mbi1JUEiPQ{ZiF1Le}6BB24us!Jf;E6isIGe&DJ_d$N5jRtaBST~!L)-RL74gIXk%-3U|yFIGoX=5$Bpp45<-@I zD1(`jTG{)xHhtfV=4$eM*2u0O#5<0mQ!a6c1S_@miG_Dp-M^9?$y{_+Bdz4I4NZTO zFc~}Py|DCZb{EKi-Mzggy^UL?YZbR^*ZpfFJOdAYCFbh(P&cUhFTdj_SLMRpnny-R zXQ-Z;Hz~fa=~OW=7^$&opcP^CnyT6*`Zi?j@jcZHD)G8icK-M8Y$Ri{dZh4uHS0q0qF`8ov_$- z0IBPyi-d~xc5Tk-Bx>}nc;q@!lz;~;yxChmveVB#DhV&M7!#imWdXZBEB(7*D(!!a zqvVL2r@7a2a+~2yhPcgCFciVS2mn|1@3u0LZ*;n!#p#tfMefH?t0N!nMa%QjP*zr!mj2bcJ+i&G*siq!pm@wo z^DYpSWY?a5AbF(ZPeyC*HE%#Yb z0In$X^|4_QPm@QR|E@dL%_=!)ab$T2RQ6!CLEh`f4iL$vsXMj?M7Lf?M4T^D-~=EG zthPxE^)^bR=()y4-OL*c-nDw%!d-8HB@5=N@xqFVGs@Pt0E_iIE>({QS#jk7crzf$ zSsh492fu$N37!wo^7i)rIoqcHC?z;&klX_RYi)Y#hTQRLYu1un1;6@}bFAWbv=Xe_ z1NMM&Pg?83Sw<+|(eV4njyNhBmcli4d7pKnjY>U8Ol&xJTV0xlT4{e-pjNnlM5}Z4 zQ$uyuqV-(__r&rXeX34q%0@|x<$~hwxT*DXmz4bc%eq5MB8S;9=LeH(rSKv^FCnWc z_bp}ErLdo~CxwKBa%L{KW<^B@KJdD&2^a!UAB_^K?vrZT_kVo;d`CO}oZZ)QL#Aiv zF3~Xv_eFT$KHM6FfD>YxY_PVZdUX7*7=TMREpIfH0!7<}3m5QNUWQxs=)u-p37|AI zbq7k!kGfk0uM$E#dQ%hs0P%s)hH!l^=x2~7tUv3&QXPJYK3A7C3?RaT0nys@vrqIU z!TlgF4GoQCm6EzGT1Nn~Er`!XdL5`bIX!wyOwP^a5KQ}6{bDw5K!#iN-uwOPVq>mU z8Jd#%t4*p(wJJ4ySmQ?zIVJAC9#uX`@XZi0wg--9YZI?0V!Q<4#9u(k!dz5#=FV^K z_X;fSm=7y}R;ZS$>%V@?1#NT-@&^jJK9kkXsxgTJ_3kUq0Rl1{)^%*j? zKCC^JJK21Griq?n28LhMN)QfTZ1@env2kzty#H=5CoW$alRJFhsXR9_M1i>S5%X;* zXQ!#byqK^wGOi%}JaQl>Fl0p2Y5nCFx>=KYN9_ixb7>|m-k2vzP)}hSb{TB`>e%92 z#(+My@7~;dg~JoQpS%%NXPYUdA?4x>irfRf)`nLzG#s@Ye?B(UFoGn;u*BlS zoKe=-UB4$!x3;!EftLofhJm@8%0DTu4DD-z2Zi$SC(#VDWNxd`Y@D(?mqm+rbw00h z(BY=OF1U!1D*yhII}>^&$(Kv1Lp|BPd-$O!9!3B8-b8lTH(zAI+EgR6jVyq4%C^qg zcgHZ2_-@qiQO{fgECcz2Qnx5*|NC-<&aTm=^8}St<3`N1`DYh516$*pTa4y_Q+GWQ zy?(1AUF;_&nvQ#M`bBy=OWILbv31kl=KGlX4J0hd5t9(ilKNvIx zV@zU~ZNi>WwJe%aD6~#vTWBfNRQ3ag&^WWr869AT>t1h65#fPGKSTh)e)_CXx`->`tJ{TVEb4MXl-?TK|q`Ab7=9&)PdrFhm_=M8Q?rwH7doK2;9Df?9G8YyVU3picX52CgZBqbMn(tKYfc*r zjthCbzC|?)@#Db>mZ#xv6CvTy&UnJhS`#agpRkr}OS)l_m4`PeHiw-LROQ*xQZ9 z4Bb<$XQ5Y%`b1O0sm_<_8XwcE4=YK6Lms7SsOe<{^l)%?P5>Cb#PP#QeIwe4oEn2VFHo!}GZq4Th>MHjqQ zNl;&Na9+hXk+(n3)_VZ%NU|6G2{ar*bVY_0QXIAwX4mvOl{ZTIJ;C09Gf(9+UK0gYvsQnF;dnaGKRD0O*x`Fv<`O_-oo@9_Z0 z*WbT+SK)HnQ+G!_CZYY;InIs!o}UAe&79&08O35P6I0XS=e%7G`pF^hMzf&;a&nfb z$qe1^>?CQUF6j@k?e1H!Bp>>VC`>N;lMVEu;dry4sKkUD<9=5Y2d?>5Z4FO1$o8(5jdd8LX~{3#q7XY+t13aX+|}@~6?l(Gu=5Q86J} z+e+T&iXk<7x!EHHi29y*GEE^K zTx;8Q2=>Ib?>V`}Ay@F#qsndd<`YxrO{brEX{h0*o?nxZnRKg9lb}&fU@YK%1N5kX zcH;*CA8$BL38lULOd}`QQpvnkn>N1#AYu0c!k>)=pbp>Uf<5FDce1jY0(#DAsi~(? zI6J#Pg_A$i()^o^beVge-`--HVKTMFfdM63Ad>Po6aq==&84MX#yPKJVGrzE%CfH} z7RFq;ZYD97hH-wS0`!dEXjmj9!$OS!FPp5kmht*Ao{(@k57CWyH%6?)aPm4pex@#J z+t!v8xbzpT=$o?5@ymszkKan{)1MlWP67?0jTQ_F5DQtOMasdqDNi`nSRiyK z2;%Zhidt9IUZ7~KccCb%GDejYrCm3mUS2By-|O=KKQ9LUeyecf4==^VGwg=O3>|al RIZodtYO5O{YgF%s{TI$2raJ%t literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/northwestern-mutual/index.html b/content/ko/case-studies/northwestern-mutual/index.html new file mode 100644 index 0000000000..dac0ef0d66 --- /dev/null +++ b/content/ko/case-studies/northwestern-mutual/index.html @@ -0,0 +1,99 @@ +--- +title: Northwestern Mutual Case Study +case_study_styles: true +cid: caseStudies +css: /css/style_case_studies.css +--- + +
+

CASE STUDY:
Cloud Native at Northwestern Mutual + + +

+ +
+ +
+ Company  Northwestern Mutual     Location  Milwaukee, WI     Industry  Insurance and Financial Services +
+ +
+
+
+
+

Challenge

+ In the spring of 2015, Northwestern Mutual acquired a fintech startup, LearnVest, and decided to take "Northwestern Mutual’s leading products and services and meld it with LearnVest’s digital experience and innovative financial planning platform," says Brad Williams, Director of Engineering for Client Experience, Northwestern Mutual. The company’s existing infrastructure had been optimized for batch workflows hosted on on-prem networks; deployments were very traditional, focused on following a process instead of providing deployment agility. "We had to build a platform that was elastically scalable, but also much more responsive, so we could quickly get data to the client website so our end-customers have the experience they expect," says Williams. +
+

Solution

+ The platform team came up with a plan for using the public cloud (AWS), Docker containers, and Kubernetes for orchestration. "Kubernetes gave us that base framework so teams can be very autonomous in what they’re building and deliver very quickly and frequently," says Northwestern Mutual Cloud Native Engineer Frank Greco Jr. The team also built and open-sourced Kanali, a Kubernetes-native API management tool that uses OpenTracing, Jaeger, and gRPC. + + +
+ +
+ +

Impact

+ Before, infrastructure deployments could take weeks; now, it is done in a matter of minutes. The number of deployments has increased dramatically, from about 24 a year to over 500 in just the first 10 months of 2017. Availability has also increased: There used to be a six-hour control window for commits every Sunday morning, as well as other periods of general maintenance, during which outages could happen. "Now we have eliminated the planned outage windows," says Bryan Pfremmer, App Platform Teams Manager, Northwestern Mutual. Kanali has had an impact on the bottom line. The vendor API management product that the company previously used required 23 servers, "dedicated, to only API management," says Pfremmer. "Now it’s all integrated in the existing stack and running as another deployment on Kubernetes. And that’s just one environment. Between the three that we had plus the test, that’s hard dollar savings." +
+
+ +
+
+
+
+"In a large enterprise, you’re going to have people using Kubernetes, but then you’re also going to have people using WAS and .NET. You may not be at a point where your whole stack can be cloud native. What if you can take your API management tool and make it cloud native, but still proxy to legacy systems? Using different pieces that are cloud native, open source and Kubernetes native, you can do pretty innovative stuff." — Frank Greco Jr., Cloud Native Engineer at Northwestern Mutual
+
+
+
+

For more than 160 years, Northwestern Mutual has maintained its industry leadership in part by keeping a strong focus on risk management.

+ For many years, the company took a similar approach to managing its technology and has recently undergone a digital transformation to advance the company’s digital strategy - including making a lot of noise in the cloud-native world.

+In the spring of 2015, this insurance and financial services company acquired a fintech startup, LearnVest, and decided to take "Northwestern Mutual’s leading products and services and meld it with LearnVest’s digital experience and innovative financial planning platform," says Brad Williams, Director of Engineering for Client Experience, Northwestern Mutual. The company’s existing infrastructure had been optimized for batch workflows hosted on an on-premise datacenter; deployments were very traditional and had to many manual steps that were error prone.

+In order to give the company’s 4.5 million clients the digital experience they’d come to expect, says Williams, "We had to build a platform that was elastically scalable, but also much more responsive, so we could quickly get data to the client website. We essentially said, 'You build the system that you think is necessary to support a new, modern-facing one.’ That’s why we departed from anything legacy." + + +
+
+
+
+ "Kubernetes has definitely been the right choice for us. It gave us that base framework so teams can be autonomous in what they’re building and deliver very quickly and frequently." + +
+
+
+
+ Williams and the rest of the platform team decided that the first step would be to start moving from private data centers to AWS. With a new microservice architecture in mind—and the freedom to implement what was best for the organization—they began using Docker containers. After looking into the various container orchestration options, they went with Kubernetes, even though it was still in beta at the time. "There was some debate whether we should build something ourselves, or just leverage that product and evolve with it," says Northwestern Mutual Cloud Native Engineer Frank Greco Jr. "Kubernetes has definitely been the right choice for us. It gave us that base framework so teams can be autonomous in what they’re building and deliver very quickly and frequently."

+As early adopters, the team had to do a lot of work with Ansible scripts to stand up the cluster. "We had a lot of hard security requirements given the nature of our business," explains Bryan Pfremmer, App Platform Teams Manager, Northwestern Mutual. "We found ourselves running a configuration that very few other people ever tried." The client experience group was the first to use the new platform; today, a few hundred of the company’s 1,500 engineers are using it and more are eager to get on board. +The results have been dramatic. Before, infrastructure deployments could take two weeks; now, it is done in a matter of minutes. Now with a focus on Infrastructure automation, and self-service, "You can take an app to production in that same day if you want to," says Pfremmer. + + +
+
+
+
+"Now, developers have autonomy, they can use this whenever they want, however they want. It becomes more valuable the more instrumentation downstream that happens, as we mature in it." +
+
+ +
+
+ The process used to be so cumbersome that minor bug releases would be bundled with feature releases. With the new streamlined system enabled by Kubernetes, the number of deployments has increased from about 24 a year to more than 500 in just the first 10 months of 2017. Availability has also been improved: There used to be a six-hour control window for commits every early Sunday morning, as well as other periods of general maintenance, during which outages could happen. "Now there’s no planned outage window," notes Pfremmer.

+Northwestern Mutual built that API management tool—called Kanali—and open sourced it in the summer of 2017. The team took on the project because it was a key capability for what they were building and prior the solution worked in an "anti-cloud native way that was different than everything else we were doing," says Greco. Now API management is just another container deployed to Kubernetes along with a separate Jaeger deployment.

+Now the engineers using the Kubernetes deployment platform have the added benefit of visibility in production—and autonomy. Before, a centralized team and would have to run a trace. "Now, developers have autonomy, they can use this whenever they want, however they want. It becomes more valuable the more instrumentation downstream that happens, as we mature in it." says Greco. + + +
+ +
+
+ "We’re trying to make what we’re doing known so that we can find people who are like, 'Yeah, that’s interesting. I want to come do it!’" +
+
+ +
+ But the team didn’t stop there. "In a large enterprise, you’re going to have people using Kubernetes, but then you’re also going to have people using WAS and .NET," says Greco. "You may not be at a point where your whole stack can be cloud native. What if you can take your API management tool and make it cloud native, but still proxy to legacy systems? Using different pieces that are cloud native, open source and Kubernetes native, you can do pretty innovative stuff."

+ As the team continues to improve its stack and share its Kubernetes best practices, it feels that Northwestern Mutual’s reputation as a technology-first company is evolving too. "No one would think a company that’s 160-plus years old is foraying this deep into the cloud and infrastructure stack," says Pfremmer. And they’re hoping that means they’ll be able to attract new talent. "We’re trying to make what we’re doing known so that we can find people who are like, 'Yeah, that’s interesting. I want to come do it!’" + + +
+ +
diff --git a/content/ko/case-studies/northwestern-mutual/northwestern_featured_logo.png b/content/ko/case-studies/northwestern-mutual/northwestern_featured_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..7c1422f32b86d6f3eb930b33467899db662b14e6 GIT binary patch literal 10083 zcmb_?byQrf(3UQ zzI*R?Xa2bFy;*DKtW{^%sk47ut5(UXFf|o99IRJZNJvOH5P4~hCp!C7d@+!p-pB@N z`%i@2Sw`3St=&6kH>e{FNz%g33IDcrG+R^hgX?H*&Yo0U@7nE2z%?P zqG|4FZ7yg56c+=CxC=cAu!T880q(Xoa3>*mQQ$x13O&{TYGwxl{=wpGEeianQM$@% z0I;1S48RNGV>9RE;Q;Upf;jm&cz8MA0Ju0f`Pn&m**W;wI5>qk1cbOa0sne{Pud(U z-U(?)%l=E()0ZglgR`@}5Ieh@n;XcD8)WDBo}E)rP>`L2i=B&$?TLfU$ph{Tb!US+ z(fut!8s=o~Xld_kX$J@VB@t?7=i)31e6sYPF4)>DEB{+C-05F|dI}l4JJg9yTb40M}ChIk;e4W@emb zFu}j&{3kp=m>(=H#m&Rd&&SQlDI*}rFUc>>FU29n4d&$3xPX1L*eGIzja$a z>HZg%`@h8sfgNE`XFEqtJ3E`ddx6>qJ7+tm4|etdF!-;m^MZH*Ov+GmOZZ>S%zvfo zpPoy@94%d87P5|Zwt#=cTFCN0*ulfe2^N&(<#}@9zwqIoSd0G@zkgzR|4S_UlQHan zb(H_3HXmYdgAtBL)LZl@%-RJj>&`pV^=3Cs} zc~#9Vok?NQ=FrEC#;VDDnOAE^miGX%4D?$6^89OD_Xs>D&twkSJ0{N3SV1Gm#J@0A8vJ8tN6b+x-G)87o)+0tMB zQT`u@JCk(0JpI|Y+TFQyH`#G1ZiOaic+wBSCKQnZBb9XbWam*+Xo&DgH@$G+uyjGvbp)mK>{%T^lS ziFIH=Ir4QD$$r1=nwRRQ{(jowscVT!gMc@vshxHxZTb48k;!hDZ%Qj!X#>aZetl^;s*^y8+UN?nf8@1{*Ut07OR91) zqq}mbH#9M38{Og2#HA6y56g>ZU-rkEp&jtprH z!Wv$-!#AZPAsY|x4t)!u!0&7owWw7V$JhN|+_||5=yXsg`%RT37z1x&DkQsK+PMKrd?f(uLQH3uzBxHF|(4YT2m6P9pv9sk4xg2 zHz;9=*VGxjnHG;B5w{V@0EGp8l*m~vcNpqU9LS=I&b~L%9k%E`;=>IfLd^YzgD^f6$s7EC!e8l&^G6c?u&* z?{;62#g@K0EQ7H5A?(E;;<4QX3rxpOa7e4mAGRZuAt}ch{Z2-dk8OE~pz5RTg5311 zicV5||2}XGm(m$wnWN1BX~N)Fxot+gE?PzhYK5x+RSM;(wn4ep_U;ig&9$k7WR<*l zd_)3ghAZu%?%JTRd>a|KLg$KrgK1t{SBs_+t2+g9&1oax$6yuT^uqh4)g2^b85P8d zAS6I3s@ts@<8E$AxW9G%Mn!JRhn`Fq+7%X~V+$GRXAN;5!bZlB?Lu4Qo&V+R(38K# z#Vpaup}V(%6Dr?k-{50WD0_hVY*oQ{tRCVt!EUrOU8yCho}!Si=2bHLs)bgXC01e{ zD+O!}Ekbj3YY0I0mT4)iO4QMmf>vs|7L;p$iy0{8Yw9;d1)--})Cq$F@7KWUkOn2j z8sT)7V8a4pCrtmC0i6@k)xNfRBk@3ofxJ&lOzrsA+g(DMKHym3r-6ThkEuUIO_h=}eT*JS5YghN znJ(S45C#?B;EL=?zlgv0uNln|t-{U0iBnM1Mof$||>%SlHQ&LX-A@L(+t&z+9D)iaj zwwtn*@xYoh`#m0Z;M4((kF`#>9h$*X zxWrLpqa`}g^a7+;yUw1|vGGnvHW-wT(&bU-QpjHib(3|>k}r4a>u3AAdPg#c`^u`q zEg{h$TbEadQxq58m!~AY;s?TXSrj+kS!s9cS*p_VmyjLMnEviZ!soT+4|TaJI0Id$QO$1lGsi& zreV@YdxJdJ$;XGzZ{a}eZ`AgutWzGX}2lvdUiQX+IJG82Usds3L!@LxiK38KR;-4;SKViR;4)H9C zw>9X_&wtNu+$>wHbtvrk>Pw+~*5g*d6kVv;^X^X=*g=bULG)!%dQEH^-3!mSK;n~( z+qW!(B%{f)mH3w1)g){RArx(nYrWYK6^Ef&Z`KuF9C*4=@@_JR!05Wb!|%5!7kw}w zv;8zJw7V6W1PcJ~v)gL7*CV(uaZvozt3M3&(hPeXWaQ99k~iO)?k4lUx0&<#J(G~- zuiu65yHI6k|5#Vxd;CyAdi&0J_~3E8K6-_wXreF!q*Go0J0Mp>;IL2HX>*gmB1LC)ysGEJ{DK|H

Gda%H|g|!l%DjX=<#=S4tsgEo8jCG$HK^ zMn!b>E?jUYKoW20BAWL#CoWgdIm;iNO(8xtfQU|Q)BRj2KufCbL ztj4i&uoa%p+~SH8unNq6F|+#fE`1^U7azJHcUy5a^as`{H@m8CHI(0bxS!2t(E)F~ zX5z=0Qd`UcbIfOhRhoqjD<6FC+IgGPR#jQ1M!lq@>axu#%B{z*gW#%nmEW>hS@}IL zZkRoqRFza+ltxmVd<7CyTOGD(O&gX#J}zgQf~gVE_ZJzVwmKU95vFM)20tdrH-gEc&v-D6W+Y+E4^klq(rBv z+ZQ|eIqU1OE0R?tXX71M`y6CseN=cU1}BR(QV8N=4o zVPR#<{P;9|1$*oB?+Wzks_(ScBJXo;4CG@3zw|VJwMk!5IcaM1Fv}5S_IAXu!fSYR z?Ze`Z49xlRYjkV#V_#8V?*!wZu08!$>2`_v>xqrV7n1X}8wEq<9AQ()bQCVHf+oMU z8rB#88g&b9;$Q{g65`Q!HhDd>OFQ9itQBye9;^L9K>Smc1jKH7Clp)pFdL_gHA5JN zEAxWbtCRaNOc4DztoL|AH>+MBtz4<^Hdtc6by;TzPZ<|KrfS(XI0MGY+EE5nUnWt- zj==iHp(%2H62!gDuHyzJ7@I?wsljf4n?%IgNel$Z0(@qKO;Y(Lc`@4JQyp#$69IT` z?<_;*i{cgKUiy*@UTtI;jHC;UpB92WL&-=S-vE@Ua=Lbq$*mW1!{kEOID$54C)O8T zpQX-MTN`LqJg(ip)+t*+HSk%wYnrl(4##~D8y_JpL!&^U5YKu%jd><28sfGwOxM>t z@Mrz?kfFfc0Xeu=Njx+E+nlGgOEJ;NA9{tkL)-5Ag=vO{Sdqc~-Zd7V^qlW2=o|z+GB-j+ zW~z+_>iNPxt@t{rnHeg%$XCv<4$>0C=f3CsVCz?3>&WHfeql&(f;WhD0>^SA0!M-t z*roj=S&PBchwc04iz({-l9<)O@j0_`5YR_Rz|Hz^PI_M)mgVr=hYqw$Vn^Q}TSd}E z;MgwmHeCL*7=u&!3-8?0(0kz~h;F!on`8`QB-YOSO8+DG)Ed&spUs{L&o&-0VPUts z?{o;rqzZezsb2 zg~~rt4}7jmQH(roH89Z7Sbp4rtBPehE1njMQz$2DR<_4C^PTB2 z?H9cDDk7=@F~2nz(^T*{ILMsfF(?)>ShWyk|3mO?yvu&*1VD8DYI>wMB(QVNfypL& zm5bI3pYVkKU7%fdaBcqR2cXa`sMZ+&Y^CZr>!=knn;zv(p^NpjKCO(g^I~-DRzCP) zd@iXG#BC1gHtp_%?7&}u7GR9Z@9~QMysWMEn3&~_C=eHP?mR?SjY_*wPWnQ^mA}Z1 zx4&*EuPLV~T;dc>5q58L6g;%%x7b$OfX20?Ssw+>6Lx#h(a~x_eY1`5Z)r=t2kXmT z#E-N+kA)|U;_W+Ks(jPlBr4UfyOo~5ex@qAyJH!!EeC{VFXc!+h&FyNs%D^C;9Iy{ zXzs8Z?rVsD-b5qh$|ypQ1d>`BB9yD)DE{sW&%zL1Za6Si*!3J8%@X_KQ+JObhf$yvEgm*k(!o_#{7yA~Rl z1+{3@ucI`~!-4QBE%x@M297V&TIQ+$DG#c~bjXSw^{;C?=$>d&Z}R0VQocL!_)P2f zP|@Y=~Qsv#8k=t^t-w7QK9U){|%NHX$fy z<+2+{OLk;N19q6J%C~N!=e-mdO1g2m#Jmvp`PEr9wP51f3mXflw#iGzK#calVqO@) z`EiiA{8w7j#;R3cBMz?_LXIa$-02nSx88IH+)Ku-DZL#B2^lSBEEsNMW6>^h*A)Of zn53oGMM8cJ$k)C7a*j3rPH@*=0)t`~1#5@&Snl>%E^|{g3QyrOJkT?eh@VoG2s*3Y zPFmLEAC2Dz!hk@oB?lW2jIo$|+m>~vEk6rX*Q3w2US8sl%buoQeDV4^>cDT=8bogk zh&G~M>xz;77>a_=13viHY-wIzW=rysZ+OdD3Jt9;+^v_HrpD?j`?Wme`nq&+odB$S z{WLQS08~#joPFrW2qUy+q=~q%ek}|mEMIriMO(ITg8dE=y4@|LiyV-e5J`^sNdnWI zMsr~TX+O4IN`nw!nU;Jf18JNbYtl?`T0_4>=agVjobEEkuVhj^i0_X}z< zSaE4UsoT!}OJj6i?e{>h5&je;FNQP8hig?CY}-p#QaL@4~U>)PK< zEx&|JNlEL*G!*}wFw$R=FWO8MGG4hd4f`#|hIt%CM|&UU`w~dp^t^(BDuZH#$gX<} zUA?1R>L;G@BO4QC2|Q9Nlf=vT#J4M)15%O?$du}bXJcO2Mla>L25`#AlLvu(gK;WIi8O9i6W1e*%asshsOJ@$)h)fK%)ite}<1G%+UJfLm+;JmDwsZ=LZ zEep!SEF266d)9Ln77abq@MNUJ#v)22Yv()PTPk|r_J^4CJnywn%4ZXAwvlSPMTfrA zX)9H!PA8FK#cNH4;Gtc=qxViiq6Yh`U$QFzvs9pvh9vhPl}wTA){Q1Ve^~}E%w&^# zA8%r`EbH0ONrxAB>clEo(@;4~tb>8VUAp(o4QJ$u#rA^&^60_E11KkbRUdP?)p#Xx zMx>Z}JqFY#_kh=a9ak#R;Y)w+;5}cHLR1FXxI3xz1$<>faoM`0kgXj_-8F>2TUxo< z-(F;qo$ew*!Bo;3`z&gq;`TM}m_#`2zhOKGDKW8ko9C_1TQ zyWN#TQBA%X3ONX+-9zDT^ttt4*Dw#Q9TK@g|Ni+W>{vP1pg}ndJr*jN=1OyKzt160 zZe3zi%{cZcjnD9SA&N%FX7T&&Z0gW$t8Rq?ZIak>15Q^}nk%L8-NX-3bk%PQjlRbt z6bs+sjb|kgwdaPR0AC|dm52Vog3GVz3k`=sOKb9x(f${1AL8Lr9*!{ay1a4_Q!Fb ziLUivwzdrbW6C<=8dhZ-YKxV|C1p{;x}}jxePg_=G8ZAZkLAxiIQ&irg0!x=i^vExWRiAPTTqHaiM zZ)1*>fKD-Ch{~9J#E>gYT4RCF^DJ-i!}K;uFdCRhtLr7@l37iI+-$|H`8G98b=ZA5 z^5k=tAJ`(^1`CNoi%ae`f}!0WpGb}l>u!+TH6=cd$ zqXXs%s~YvH(j8Tq;+XV^QG)4%s2sF>sUH=Qvsqii;8kbSJb~4Iu9q2zDQNJyGbq4h zjeSHWCCl6k)2adS=>|RPmRsohCBglxG>%Xr+t+egQtEWGzp&-EP^kl|I6sM0MP0d- zC%nd}VRRSQMKp*=UXmz_@XPLNenfD$ik!|k7)5D`5La3km5~pw>%|Efc!pv<=DlB8 z6dH^;(5XDpURk5oJ)37A0;%)O|L_17!=56RY##K@yf-rBY3&{8L-?FPGli3 zgR}Cw)u_Zk3Z*-xvYKdDy4#~1cOO3dm|9ZbyF`8$bC^{)QU52flh4g-myf2-dWVYT zZY1NQ)2X06W4sW8c`l4BkpoN28EL|TYlE20Rg>2}gY1HC1G6JDx(71jfLF5wT~r|x z%!8w-0{JLqv3zN-APyn51yXOZRA^XTDLZFQ@Wy>8)O)fYjU8mf_K z32fyYU?4X0>dJf9gL@{RWP>M=Z054EvMNkBMp_C#92(g5?pNakzmNSG1E zE@L;$q1Vt~zBz^^TtGnrAEN7<8&&=eI;Y}YL0G#RR*Q4&`=Xl5`s*H-dA!dYh0Y+7kD)?_?0STl78q-W&I*Y=0Pz zpp7&O8?{6qtL+MU{cA$$S1-L=%j)Moz6U&H9>Nopej^o@VXmC}7&mi`W{OCRPA>XW znQ#t`gfN?6rrPp)nSG?6rYdbx&E`RjS-?j)N;5av0qf}+)8@qw@+)WFw#~@~r6Y2@ z1Vd06cp=E3fL5{+>+{y+oKl0-@Ns&>_&iXo*7!raiO>0-T#&cuX-4$Bk8Y+DnX;Vt zM=>18pG`F1@uXsh5A3_j*}eydxi>9KlgLD!l`_&E8BFMZcz9lW$#4_wZBO%~j_Sge z9hZFJRNCOch(@ucr?ZrRHL%CuGW1!0VA%YUlcb@5FKRpxLJ`JtGjDIk5q`hwTB2oD z`h(%)a=@XFs4L@ywv7beZNq7rdEQ!{Qn;MW0@_r|>St{UDD$QfGM7$yz%LYn4MaUQlp13Tv zpZ7YZ9X=aEGqk%u;zeR{tf2+c8cU3hri*WV9^MXV+%FS+t$PZCf;)cxIB27rRn5nt z^e}TYT07#C32O0=nN3?X1Vkv%(?5CReH;z4_rH#O+PsBp6mNgIx#T8jO>Y!!k3Glu zee_0aZr^l&(=nw$M}$dYhNaTM{ln^t75-{gb83itPn_&w?z(Ls<1noAs33V0Kl_b( zNEj-;K0Qu5oRL6=$zXg+3RCw8sVY%uHC9XEu{UaFTveTj7cu z{Wp`tC|=RVcE6X84FdEcC{gg5`QrBs^ZVqa<(k@aJXQis@M-#2Tmy@D1Rvo!3i(JA zcm{2mS;OcGBt!`x+dK!(QdWL2@XdYXAa1tJZUc>)cwKncDzS9ExX@+>kRxrU#d_pt z&t^{)8d1)aIyftPzUMMzlxsV&sPY=1yv*1%V#m#LD>)6S>nhv|6h~AiHM_>-5VGHud4+ zPdOt|kqiU1DxbZF=oR0sf=wz)cK09lMk-|$^)@~8RC{r5v{3_MZ4Jiq68f^!al{mt zRHEin+up5DXIJ>^RVW-dr-}4l*Et=$QRifT*lyi2O#?V_q`tvUEE{5Di8Xt-P?jgSlO!8S&f_(ooF@C0 ziP{f{%z4*jKFky)Iq)1snY(XG0wY$Khns+RjE59e179h|fEz$vwENk5z~m1OsEHfI z$5m;$&+xHT?0!6laNARKewq+c@UfC7$8sNg#M$I*1I-PI^tG5J@3QLo{%kF5F)^bz z-8EFK08nLY-zH`xBpNrn5$sTat>7%S9z&HJRv)6?m3eqSwZ1ufvu53vM(f0GadA3T zYQt-Lvk#-m5FjO_g4WE(mZM!LBcdB(GQ6r5zo;>Z>}p#U?Jm$kD5E|PpuOBb@9c|i zHEFDUU*|eu{A|Ju>ko{3i5ylT8kFo$>S8ySF z$V9MHH@uW6L{)zE;C=Z1)>)QitBGdh)VY&5A+h`hwzp7@`{HCmsCavMZQfW8l+@{a zf~BZ;NydrdjLE8!5+|=aB26tn4d}!A0~u4tr_JejXz$#$?nQvft=RL&H1!Tjjnb8X z+PY4Wh>Gm)dNkv8UABp4OPd)Lwl(#>>Fu|_M2Ubbod`GZSZ?2Qy>eakQ?f8G^6W5y z0Yr7WJ<0JB-JivDB-8};OAm>piTsJ;e2xCX4%Md#-=YBa^^1|;fyoC@;KEkf6<4tT~jOyFcJ%q)B^h7Gh zWg3Tn@?1~Kn?i`q=I7{|$D9un@p>!KUK_!7zPguVS8G=6T16eY?`Kwx9M)aS8r})N zP1-{*qfs2HPc?Y~+Px}@=u?z5i8zfxFfCJbbill&BIPQ0XYOn9l)4wX!#``ezXsx( ziA0@8;9dNvG!MQhe_Q}{`C=E5DfM`0gV8VMD^5puB1@rmLoYB!aVTbQzak9x!Wv** zOKyMsaLUQ`>FXvj?rx7vB2+V4CK-xk(nyq}zJ4YBCB%F{{R%%!AWU{C=GZ#ILQ=Q* zM`T=uVRKI_r_uGHOYvVC;b zZ@~e_27N#QaG_8(1i!q{K-ld93SE8awh6389@=skfy34>u#3ckk`$O59f7UhgA=}I za{&n@I<^!iCvUTq$#Jf`UY79ZS--uvngaj1=FFD9#R}n zJ9eNQ1N6)}RtxIEaZ;u#b))6AecSKEnKxDFqE9ojVC?q{5We6>6%lTIw7kbIK#AL6 z_@`vTl%IQYcdrB&P+-CEqS!f+qJ&f<-(-?PIi}6bjy-=;^c~LK>E#4K5~))%=G?I@ z{&{JgCJ?%KS_axCnT~`9S~GJeo6rYWu^c3tLpuTkUJIiGIHXEpR)G zzHO`L(FG{vuV zvxsj(E;5&?4mj-L^Y<5UxVaN=d$8QN=xWcIbV^ZQ%X#LLT6h!#W91u8=P53_Jtr-1 z;kTYEX0zOxiz(kCNtmO;4VI!N*wlem88*f{9b0&6>jghpb^NRm_SLOEh!cx0uYxux z(I$nUTn|``XCp1$mFwtJ;>v?N&ITXF=6VfKLy;6)9Y$h*c2;rOOTAe3VmhBF*J)Qr zNi!n-jV;^c<8J?SoU}u@Gl$x#(ju+b!c)F;pFDgRdetRfTdm#F&u(G-_Ar)sCEpV~^|0dpGuq$d$L_nt11 ztq!^34D0u=!@bW9!uO7_HuY6DnY)%xJCRm%1-yQKB=X0@G!T_*V7%~XG!-G-vew{| zHQb7{He|coG3Ee(ya;DL;B~yM-kd+dy&62mi^@NeWPF#Xq&&ozmO + People at Ocado Technology have been quite amazed. They ask, ‘Can we do this on a Dev cluster?’ and 10 minutes later we have rolled out something that is deployed across the cluster. The speed from idea to implementation to deployment is amazing. +--- +

+

CASE STUDY:
Ocado: Running Grocery Warehouses with a Cloud Native Platform

+ +
+ +
+ Company  Ocado Technology     Location  Hatfield, England     Industry  Grocery retail technology and platforms +
+ +
+
+
+
+

Challenge

+ The world’s largest online-only grocery retailer, Ocado developed the Ocado Smart Platform to manage its own operations, from websites to warehouses, and is now licensing the technology to other retailers such as Kroger. To set up the first warehouses for the platform, Ocado shifted from virtual machines and Puppet infrastructure to Docker containers, using CoreOS’s fleet scheduler to provision all the services on its OpenStack-based private cloud on bare metal. As the Smart Platform grew and "fleet was going end-of-life," says Platform Engineer Mike Bryant, "we started looking for a more complete platform, with all of these disparate infrastructure services being brought together in one unified API."

+

Solution

+ The team decided to migrate from fleet to Kubernetes on Ocado’s private cloud. The Kubernetes stack currently uses kubeadm for bootstrapping, CNI with Weave Net for networking, Prometheus Operator for monitoring, Fluentd for logging, and OpenTracing for distributed tracing. The first app on Kubernetes, a business-critical service in the warehouses, went into production in the summer of 2017, with a mass migration continuing into 2018. Hundreds of Ocado engineers working on the Smart Platform are now deploying on Kubernetes. + +
+ +
+ + +

Impact

+ With Kubernetes, "the speed from idea to implementation to deployment is amazing," says Bryant. "I’ve seen features go from development to production inside of a week now. In the old world, a new application deployment could easily take over a month." And because there are no longer restrictive deployment windows in the warehouses, the rate of deployments has gone from as few as two per week to dozens per week. Ocado has also achieved cost savings because Kubernetes gives the team the ability to have more fine-grained resource allocation. Says DevOps Team Leader Kevin McCormack: "We have more confidence in the resource allocation/separation features of Kubernetes, so we have been able to migrate from around 10 fleet clusters to one Kubernetes cluster." The team also uses Prometheus and Grafana to visualize resource allocation, and makes the data available to developers. "The increased visibility offered by Prometheus means developers are more aware of what they are using and how their use impacts others, especially since we now have one shared cluster," says McCormack. "I’d estimate that we use about 15-25% less hardware resources to host the same applications in Kubernetes in our test environments." + +
+ +
+
+
+
+ "People at Ocado Technology have been quite amazed. They ask, ‘Can we do this on a Dev cluster?’ and 10 minutes later we have rolled out something that is deployed across the cluster. The speed from idea to implementation to deployment is amazing."

- Mike Bryant, Platform Engineer, Ocado +
+
+
+
+

When it was founded in 2000, Ocado was an online-only grocery retailer in the U.K. In the years since, it has expanded from delivering produce to families to providing technology to other grocery retailers.

+The company began developing its Ocado Smart Platform to manage its own operations, from websites to warehouses, and is now licensing the technology to other grocery chains around the world, such as Kroger. To set up the first warehouses on the platform, Ocado shifted from virtual machines and Puppet infrastructure to Docker containers, using CoreOS’s fleet scheduler to provision all the services on its OpenStack-based private cloud on bare metal. As the Smart Platform grew, and "fleet was going end-of-life," says Platform Engineer Mike Bryant, "we started looking for a more complete platform, with all of these disparate infrastructure services being brought together in one unified API."

+Bryant had already been using Kubernetes with Code for Life, a children’s education project that’s part of Ocado’s charity arm. "We really liked it, so we started looking at it seriously for our production workloads," says Bryant. The team that managed fleet had researched orchestration solutions and landed on Kubernetes as well. "We were looking for a platform with wide adoption, and that was where the momentum was," says DevOps Team Leader Kevin McCormack. The two paths converged, and "We didn’t even go through any proof-of-concept stage. The Code for Life work served that purpose," says Bryant. + +
+
+
+
+ "We were looking for a platform with wide adoption, and that was where the momentum was, the two paths converged, and we didn’t even go through any proof-of-concept stage. The Code for Life work served that purpose,"

- Kevin McCormack, DevOps Team Leader, Ocado +
+
+
+
+ In the summer of 2016, the team began migrating from fleet to Kubernetes on Ocado’s private cloud. The Kubernetes stack currently uses kubeadm for bootstrapping, CNI with Weave Net for networking, Prometheus Operator for monitoring, Fluentd for logging, and OpenTracing for distributed tracing.

+ The first app on Kubernetes, a business-critical service in the warehouses, went into production a year later. Once that app was running smoothly, a mass migration continued into 2018. Hundreds of Ocado engineers working on the Smart Platform are now deploying on Kubernetes, and the platform is live in Ocado’s warehouses, managing tens of thousands of orders a week. At full capacity, Ocado’s latest warehouse in Erith, southeast London, will deliver more than 200,000 orders per week, making it the world’s largest facility for online grocery.

+ There are about 150 microservices now running on Kubernetes, with multiple instances of many of them. "We’re not just deploying all these microservices at once. We’re deploying them all for one warehouse, and then they’re all being deployed again for the next warehouse, and again and again," says Bryant.

+ The move to Kubernetes was eye-opening for many people at Ocado Technology. "In the early days of putting the platform into our test infrastructure, the technical architect asked what network performance was like on Weave Net with encryption turned on," recalls Bryant. "So we found a Docker container for iPerf, wrote a daemon set, deployed it. A few moments later, we’ve deployed the entire thing across this cluster. He was pretty blown away by that." + +
+
+
+
+ "The unified API of Kubernetes means this is all in one place, and it’s one flow for approval and rollout. I’ve seen features go from development to production inside of a week now. In the old world, a new application deployment could easily take over a month."

- Mike Bryant, Platform Engineer, Ocado +
+
+ + +
+
+ Indeed, the impact has been profound. "Prior to containerization, we had quite restrictive deployment windows in our warehouses," says Bryant. "Moving to microservices, we’ve been able to deploy much more frequently. We’ve been able to move towards continuous delivery in a number of areas. In our older warehouse, new application deployments involve talking to a bunch of different teams for different levels of the stack: from VM provisioning, to storage, to load balancers, and so on. The unified API of Kubernetes means this is all in one place, and it’s one flow for approval and rollout. I’ve seen features go from development to production inside of a week now. In the old world, a new application deployment could easily take over a month."

+ The rate of deployment has gone from as few as two per week to dozens per week. "With Kubernetes, some of our development teams have been able to deploy their application to production on the new platform without us noticing," says Bryant, "which means they’re faster at doing what they need to do and we have less work."

+ Ocado has also achieved cost savings because Kubernetes gives the team the ability to have more fine-grained resource allocation. "That lets us shrink quite a lot of our deployments from being per-core VM deployments to having fractions of the core," says Bryant. Adds McCormack: "We have more confidence in the resource allocation/separation features of Kubernetes, so we have been able to migrate from around 10 fleet clusters to one Kubernetes cluster. This means we use our hardware better since if we have to always have two nodes of excess capacity available in case of node failures then we only need two extra instead of 20." + +
+ +
+
+ "CNCF have provided us with support of different technologies. We’ve been able to adopt those in a very easy fashion. We do like that CNCF is vendor agnostic. We’re not being asked to commit to this one way of doing things. The vast diversity of viewpoints in CNCF lead to better technology."

- Mike Bryant, Platform Engineer, Ocado + +
+
+ +
+ The team also uses Prometheus and Grafana to visualize resource allocation, and makes the data available to developers. "The increased visibility offered by Prometheus means developers are more aware of what they are using and how their use impacts others, especially since we now have one shared cluster," says McCormack. "I’d estimate that we use about 15-25% less hardware resource to host the same applications in Kubernetes in our test environments."

+ One of the broader benefits of cloud native, says Bryant, is the unified API. "We have one method of doing our deployments that covers the wide range of things we need to do, and we can extend the API," he says. In addition to using Prometheus Operator, the Ocado team has started writing its own operators, some of which have been open sourced. Plus, "CNCF has provided us with support of these different technologies. We’ve been able to adopt those in a very easy fashion. We do like that CNCF is vendor agnostic. We’re not being asked to commit to this one way of doing things. The vast diversity of viewpoints in the CNCF leads to better technology."

+ Ocado’s own technology, in the form of its Smart Platform, will soon be used around the world. And cloud native plays a crucial role in this global expansion. "I wouldn’t have wanted to try it without Kubernetes," says Bryant. "Kubernetes has made it so much nicer, especially to have that consistent way of deploying all of the applications, then taking the same thing and being able to replicate it. It’s very valuable." + +
+
diff --git a/content/ko/case-studies/ocado/ocado_featured_logo.png b/content/ko/case-studies/ocado/ocado_featured_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..0c2ef19ec3b0359285ba05db312d2d8b10cbd5aa GIT binary patch literal 8089 zcmaJmWmr_*)dCuXXkdZ7mfdd|G?}06?Us3e`ombErEM7ajFH)$X=N zH4hO=MhHDOJA|*bCmbLTbF+a1)m*IY;ks~Zn7_veTp9qt;B?eCLKtakNZPu&@LT_Z z;rDZKM_~g1(z1T;*0xS?1keU<@8}9)K5FY?20FqZ%!Xi1kfysL+`&;bz!R<)prvmc z;AATSW0sWxO8ZHo2)Muz)<8cOXIC#tKM3<*bR|*!Kf?mdz`sBcP7vmQF=eEw4ODdV zgag6+qI|Z3A|Rlc1izptNCYg%3lstgiV1+g0w7U7kf0<;TvA97`1i$(n#~htC#efn z{(COe8-&>bfpC`;5b*W&<@Xilck{Ft5R{OR5C91Y2nq3_Ao#reT@luPe6C(B|4@L! zy=*-l-4Tv%uE0MOt!>=A5fEk+r~eYc#a&bL--KPg{#F!9WdeTI?gE1RAORPbKXUyA z?S;^V|A&qL3GJot?+zExg?qVqd)lJxVaM_hGD>&<|3rTPQQSyE-E6&G;I0TYD1;d` z!VhzVNrFTalog?hVhVzyLV|)y;u505@=9XzLJCR}iV9$H&_B3pu3iXhS6lc$xQ_q9 z75#s4B^5p4)(AIGeK$Aff2^nN;D&JXa&U78Dk}bEEg+|+wXLJ;pFyra3jUj4xTm8J z9H#8)<^udnLz0gFg9{;$l7u)^R7gS|Eb^BGLgK+j&th|8{@4E6V3RQO+GC!#e-~(Br5< z<@Nm*_swyF^qYHqlyQHX?{z43 z3F)<|tF5)#(Xrm(?~c%k#UfB~ zSQC0zIEcpE>Q5eRxEld_EcgJ>w!C!dZStqbdNk1;Y6)BIdeku&XtBlgw6fV8F7cC6>X%=$5@?tDd_Tt znCgDDHjw zSqQnni&E>-=7nhCrlM zFL6BPY>-~+#d@rcpVt*Re&n^1G}WWuuN6c`>1-!AN7;_?Rq+Y#DBX(;kPvwS#r6#eeFML}L)j6px5C?zq<(5y?WVo&s`ewP!-;Jk!Kk|w(O zu{N%~JFMhqglWoYM6Q7tlzUkDjr$C-r>!sTBL1-sEgG|}S{qvLT zOUJmuVw48KFZX$6WI=+59kI0>9RgSuR+`JDtbmph>lVg`LRAWw(-&r^4yprSYM7!qN{a zQ@g$M-MyTe8CPQB++3Q=yX~_E>YA~sZ6QHYWAV9G49v(^mxNCs+Yrng@1Y4;64!f^ zCpy@V=JdYzu+wixVeHZgYnju|7DFVN58jz&+ca+^4RJc-^a$>Z1YWgk3&ei`-~3RR z(2uOcJf4$Sbnl;g)z`W7P~g;luP@v!!MG(^Oyr(_Rk^*g+ONRyDfX}T^8yG` z%4CkmkH1d*sBGu85y{-5kk9~KTGKHS@8P+ zHyqpuH;y$dk|&TUvEOL>tpMDJw2Y5R{0uYeR_33SD#yu37KX8}=xEPCNm@5zoG*w&mFZK~g2^HUyL7xh zL`vZX{8o{nE)J?a{Yfmfx4`y|d%ea?G;Wd-DWY6>PsWr_Jtgbe`J|*fjYSQaz9g3{ z2aHd&*uC)ckP9-5k0oZ`I-wv8#=jy%563T>VvKw_wYI}-UG=0yq?MeZ{+Zm9+7q;} z)TND*Z-wD;9&5p^4FgrjQ$rZgpBlrwYo5jP%-4B|@#k1N&e8cS zJ&Ap~eN~$J+IRYe38!@cs3)s;=XK8T$}C|=9|DoW7wP~l~>Xqc+K}FaTbS-sJqt7 zcqg{4)u{SqDu`yEbg3>d>flBVZsNA3_D^oAUbRmp1*z83f^l3+WlU}t2Dgg3Lv=l} zHH2P8>ufX5(j2oYeH!SDp1~B86RE{8^q8+2-0EZZGH%|})BR~ErG$pRi6ldp|G;RW zdGR=Y?CW~iI2EGEp}4Zdb@9~H_B8zFC02JS1y;6w{epsM&Llf-Xj|;J=E_<9 zOV>sw3TUN5h@y{)6*f;miMy8Ksm#&S8L>Mt;mdo@(fw}0%vFJJa1rL2Btur);@AIH8cUB&%Nib0 z?jvLZ;a|((8vCfbHPf{kn(@+#0PXzkB{8Kxe*o9O)))2#XXftVP5R!&mWvf`BmCul zr}|s9FHb_dskQs-7UUjK&(+Ml$28Lo!BMz3Yh*WKU9f4GA!yh7c?7&=`QhYkU*Y8I zGhD#_I?;!AdBE<#jLDM!!2sUXx^#7JDc728LkZG~n@O$BwwY6p6fm?Ir-$vb_0#0C z&8Vs|H?<=tIc;#Rdyd%}doFD8ggBShu(i$k^_ zVU9~bUu|4i_I~K+!(VW)H!doFJ-8QrS8t%$fcWhtRutZsO32X8qkc~J64o+61R!1; zUKgJyv!J&VuJ__2RY$uo;W)x4%04hrFO!|@-HQ%s(CtHRKUpq*0I)InCiViuz+)a+ zUAxULUPez!NZjuGJ*B8;w7gn2ys|psV;5J4y@?=a<$ zU0{288DHxK)b}35V;2)Hic6o@vI1NB1OpVCM$9&1e05Hyb7{9CV#;;W$V##UPZwFg zeg6{0hobRpX7wFimvDBRvrhv?ybC|mYdaYMC_h6Xez1s$163xAS7lkXb4=$~h$YrrHZ>yoW6T>+Du`cBq&sVR*Fd$cvwB_j z&RzCDP~X!T>;fc8PYufy?CUIE2Zi^h|Q*oF6- zGnIIu3`AYZOqt^qPebRbAhR3b^73niqPDRf&a)@0DWyBT^;mLB5F>)h_npNy5=+TP zh=Zfca1Mnpk00uZ)j)YGJaLgK@)utK8)oK%lnj7Jd@w`o&zO8 z@cM~JLnHB$O@hLuoCregjzuGtGU<==cH_^P2f#E@b%x92MgX56R#8HDI>P>}D+A%OB{ z%(Mq~-0@`Fcz9&mQv9Zn${*AZY?kA5>rtF`V`=r%Eyj248WXo^%pwNY>^l` za7<#UP@GQ{=K7CiYl;!2(WLXNosiTvdQ=2wAuxNAV9TBLSW3xMRo4lXx$#mO zJmv#2jvM(5*|vC)pi(y>-nZl4&ah{Re9xJKAM2DY`AR(EQwA{+Qzi5+O(b1K4814B z@u0Tq;cVwq*a#os-1i_m)~AW3iH7jXkM4*(XHvGFj58Bb=EUunLz!fOCL^D|EAhj| zPZX$*j_MG{V23S>M=_G1SfGYlx!hvYL-9m?Yo3cersT5bSWq~wmvbNffa<1LS7xX6 zH-YyCe_Ba|&sU|_hch77)^mplspZIeBi_62UtzLRGriIZRfahZ*?m-XZfEn5Kbg8a zA_A+3l;BU+F0MT28QAY!M`L0_KLQgO|8ZKP%cabeVQHfEM*4j6iOl*>KhYeQt7TsS zi(~G;KJ33*e%6Q-ak0g`jy4LA!=<*q9LO7^iT_bJoU7ZxP^v9Q2%)ejSuWs=MA4NFRj2^oY~OG87W9xV<9n*{_v_E7V% z2toUxUpD=j*Q8v3aIk-ngq~E^y+D<&a=#mV{`^7Ycjx8D&$BGT5npxj`q6+ja zLhi(*eBt)?7}*Q_kB77~>X&{6-F+z6dz1w+E!WPK2z32r>V_2yWmVJ42CbJ+zd4H2 ztogp_vQpnzP)iDKXf#Mj@sau(U@`gN&gbQ);NQ7dgyFUEUxck z=lParqQP}@^C}r$bXYG}8i%ZuZ<1;0{gq|D9lc~fykBvXE8C>$4BW8L9&lh1F1PkD z`o~0msPFPMdm7&H;?rgT6(5xo6ngayivYWSIa+7_MIAl?HD7R^6}hyXRY8|3k~j~q zk7Sw_o3e8q6oLI{5PSPa@)Oyar^%5Wfliwh4L>GvJ_yotq6uMmgDwh)hyf6Xu{D;U zmYwM*80gSycjH=^nz3<9nuIf}TJzIB9mVG#3V=w!V(-(6*53hF^OX2#YVnqVMejZi zmy=iYe6=MX6CaX~a>vic&YODHZ(UeeNJK0yaw6j_d8k) zJSHn0uV`LXSJN5{S)bIktg-kA<3d1o*|%Au&1AesU!4^y!h{~uS{;*J+QZB`J+a&| zZV4cr8%u81UwUX;XIc&;&+RAk2yk&7`a~pP!zm+a>=9kj{KF9ot$tiyN1ed|9go%; zVe1NpxV_qZ{6n?H6*&Po zRets*OWV72sO z76BbCU4F>c{W_?BsNiCg+LA`hhc|y`%txkqJj3G{D2@@tP8c&0rE3{<{w7z4lQ#@kRq!?-T1}=x1m?ql8A%WODSk7sdGrJHH(eB zxCVE+i(ja@x_X?ca`^UDi-gx~webOdbZO~UO_Q5SDgB^~B=@uBHYZ?#kJBwi`Y2XS zJNhBo7$eCz=I-s;YJU9E$A|2LI8Jjb5a-uRGtA}ClbEtMH>VAg+0wgF&udJ(dLAyw z+1}TkFvPA%Do%YilwL`2obL8l^0#rfcpYx(a*thOB9(A7ThpoB!$m55JQPVHz1D(@ z8Ng#$C70zBw%q&81|(LE6DEL5{{n;)^WN;Io%WJ?+_i;OgJMLadDlYchAebxs<-B- z>MCc+c)bEmE=|O*Hs+=)QuYz{TLtmw8R)dVy$Xu%PCKAv%o7JuHQTSovb7?9ExJepkrnH1-4> zOsTNpP4t{rq~c&9KmcqB@mQo{Vf5^2l;?L4xHvx6B3R@nla^|r9Ktnh%Fz~)mL}@%*aQRZZ ze8oY8i5U|@e%LknDS~W${lyDBKobKtK6q2+6M~_b^+z&LCM#L9dmpG5pTwF z>G`L1#dlapCNEBC@y=+jNJK?q{M)DPCx1!dpAy7>ba^byqC$AtS-{Op7(oIPY&0>~ z(@P^)1li##l3L>UcYHi}buy|B6Dw3FBU^t^!-4$;ruwWR&r>mmRB>wTQkw@r)i1d+ z$HB)I?Ku?Py#NM-9Xfj~-Ay~@vR;Y5wgM95S(@p7JL1Hg62JZ&$PD#(p-?D?zsg@9eK5$T|K8xbfrOUk`*fpSI``Q7^Dlb{tr#?`ac^n}f(^ATAJBI8 zSV3j*OwOU7UuX8Nduoes8!hhdbnrrw73Ks_&m*{c5H?PRO$$9f$@aSzCV=In)gbd5 zX4fxcrs4^%Xvrh>D%XguBK?;e|#5qR|vxQ zF4S4x&XS4mjk@xK+FD6Y=AEwh^b;i#0fZ>b2%;5Abi+GwR7ho*Lv_lg!A>Hb26#5l zTE{0VtIHh)Kln}VW!8LeJxA+($}PH4fo=nMN<@^1Qtb5!yjilR3$t;Uqb?N&f0|Bm z=qHcHjcb$;8++_aQCd>+_Atyb`0UE#h~sR9@q)}Ho&paqvfpLT{~N%=Q}^Yx+|hFL zLpC976mc}%4l@z+AvfUKG)^=*n;v}uS#;Y_ z5IcAYpNoHP2kkH#<7SIO65mv-e=j!!zYLQs1wRfvnM%Rl{A&tH6?rt5HrT4Qd zd2b%kB6nSTkJ~%<1!$9f9s-J|_sqNeZ$7WS{lrMt7<9dQA25uSu(K*_tu3AH$z15ZDQjS;ZxAq>m1$Xf*GHn1 zp!NEKB_{be$a+LT8~0?YDl3c?W(;NioPD`+Z8t>RJO{I`CBP@J41zzxBy!-U(=7ED z=osE;6gEoQ@AE$;t6#4jEG^Ca(Pd)v{jTT8{Vgh9=U?uk&w0zFtAm3V^;wG0NcBui zn%sAgqw1IYa1gE7-D#0Pv+H^8AmIneqms-7WtNI`gR)S^yC8=oQ{iWIMu6EV$~V(G z;VkkT3QVJp(HU0v+C__PH`Pq z3z>X1tw?nky|mHOr&xgKfW<0%1X*fbW0OqqcEWo zUH_YYwO3pio@eu@fA{ExiX4s&;aYhHf$PYU(2$#v%fQeu0UcEw^H%2_xV&m1G0q$kBzG*kOmJYLBlS7e3X4&YZ2pZtNq!#l6b1Rwu#Y|tIsOe7mNU+ z332Q%y`XL}D|a;IW9rQ`A+?$>vjQ@{9lGd+NP?O%3?m>8MM(j(50o~{bzNZ87kXV7 z5g9r2tTBeyP`lplATZ1vI6*gy53d;P?|+q^-a|)XT)oufw%QKo(J(2~l4_ttn?SdZ z3Q!;&_Z}g8z7ptqm}@L-_Z3euRK?wT@3$(96*Lr0KuvaAAZ;!3z+pd&etwo`0@xM! z?PtrR^u~HGb^;D3Au%y}K7$A_I=3mv&YoLn^h2cEOKcJ?M#(Rf(V#f;p;b$Bb3Fr{ z@TMHZ)Y?~UKp`Q0d%%o(v_{CqjsL3`L}v36H&N4xv=Zpusp)x%VS7ymsvPtevpJtw zSXh8sD&}O>y5XS#w`0ebilm^hI0gc`fv>H#%`_XrQp3WjKvK|oq^K#XV;{d+7DR%=p zp3qc|bzRqVcHaMLe|L4w@g~K4G!Koc-G1yfnul+?p}qoWozOV9a??S(a8XVNwS5|T zdU^3%ZVCF&$eO(KWY^LQIaF`O0c?ethz)ouB&SG@d$x~Jl)+XMPAL9&g*2Q^gNJq;~> zT;$9$xKOLwfd^_Zs9$eE6e4Sm6vnP)=6BA;O zYYFi15_t5nD==xl>;|~=GL7j+S0ZBw!;RN9o`?wY^Y(o8rR1$L;_bA4e?oA=pMy5ee)F(@%6Q+fn +

CASE STUDY:
Launching and Scaling Up Experiments, Made Simple + +

+ + + +
+ Company  OpenAI     Location  San Francisco, California     Industry  Artificial Intelligence Research +
+ +
+
+
+
+

Challenge

+ An artificial intelligence research lab, OpenAI needed infrastructure for deep learning that would allow experiments to be run either in the cloud or in its own data center, and to easily scale. Portability, speed, and cost were the main drivers. +
+

Solution

+ OpenAI began running Kubernetes on top of AWS in 2016, and in early 2017 migrated to Azure. OpenAI runs key experiments in fields including robotics and gaming both in Azure and in its own data centers, depending on which cluster has free capacity. "We use Kubernetes mainly as a batch scheduling system and rely on our autoscaler to dynamically scale up and down our cluster," says Christopher Berner, Head of Infrastructure. "This lets us significantly reduce costs for idle nodes, while still providing low latency and rapid iteration." +
+ +
+ +

Impact

+ The company has benefited from greater portability: "Because Kubernetes provides a consistent API, we can move our research experiments very easily between clusters," says Berner. Being able to use its own data centers when appropriate is "lowering costs and providing us access to hardware that we wouldn’t necessarily have access to in the cloud," he adds. "As long as the utilization is high, the costs are much lower there." Launching experiments also takes far less time: "One of our researchers who is working on a new distributed training system has been able to get his experiment running in two or three days. In a week or two he scaled it out to hundreds of GPUs. Previously, that would have easily been a couple of months of work." + + +
+ +
+
+
+
+ +
+
+Check out "Building the Infrastructure that Powers the Future of AI" presented by Vicki Cheung, Member of Technical Staff & Jonas Schneider, Member of Technical Staff at OpenAI from KubeCon/CloudNativeCon Europe 2017. +
+
+
+
+
+

From experiments in robotics to old-school video game play research, OpenAI’s work in artificial intelligence technology is meant to be shared.

+ With a mission to ensure powerful AI systems are safe, OpenAI cares deeply about open source—both benefiting from it and contributing safety technology into it. "The research that we do, we want to spread it as widely as possible so everyone can benefit," says OpenAI’s Head of Infrastructure Christopher Berner. The lab’s philosophy—as well as its particular needs—lent itself to embracing an open source, cloud native strategy for its deep learning infrastructure.

+ OpenAI started running Kubernetes on top of AWS in 2016, and a year later, migrated the Kubernetes clusters to Azure. "We probably use Kubernetes differently from a lot of people," says Berner. "We use it for batch scheduling and as a workload manager for the cluster. It’s a way of coordinating a large number of containers that are all connected together. We rely on our autoscaler to dynamically scale up and down our cluster. This lets us significantly reduce costs for idle nodes, while still providing low latency and rapid iteration."

+ In the past year, Berner has overseen the launch of several Kubernetes clusters in OpenAI’s own data centers. "We run them in a hybrid model where the control planes—the Kubernetes API servers, etcd and everything—are all in Azure, and then all of the Kubernetes nodes are in our own data center," says Berner. "The cloud is really convenient for managing etcd and all of the masters, and having backups and spinning up new nodes if anything breaks. This model allows us to take advantage of lower costs and have the availability of more specialized hardware in our own data center." + + +
+
+
+
+ OpenAI’s experiments take advantage of Kubernetes’ benefits, including portability. "Because Kubernetes provides a consistent API, we can move our research experiments very easily between clusters..." + +
+
+
+
+ Different teams at OpenAI currently run a couple dozen projects. While the largest-scale workloads manage bare cloud VMs directly, most of OpenAI’s experiments take advantage of Kubernetes’ benefits, including portability. "Because Kubernetes provides a consistent API, we can move our research experiments very easily between clusters," says Berner. The on-prem clusters are generally "used for workloads where you need lots of GPUs, something like training an ImageNet model. Anything that’s CPU heavy, that’s run in the cloud. But we also have a number of teams that run their experiments both in Azure and in our own data centers, just depending on which cluster has free capacity, and that’s hugely valuable."

+ Berner has made the Kubernetes clusters available to all OpenAI teams to use if it’s a good fit. "I’ve worked a lot with our games team, which at the moment is doing research on classic console games," he says. "They had been running a bunch of their experiments on our dev servers, and they had been trying out Google cloud, managing their own VMs. We got them to try out our first on-prem Kubernetes cluster, and that was really successful. They’ve now moved over completely to it, and it has allowed them to scale up their experiments by 10x, and do that without needing to invest significant engineering time to figure out how to manage more machines. A lot of people are now following the same path." + +
+
+
+
+"One of our researchers who is working on a new distributed training system has been able to get his experiment running in two or three days," says Berner. "In a week or two he scaled it out to hundreds of GPUs. Previously, that would have easily been a couple of months of work." +
+
+ +
+
+ That path has been simplified by frameworks and tools that two of OpenAI’s teams have developed to handle interaction with Kubernetes. "You can just write some Python code, fill out a bit of configuration with exactly how many machines you need and which types, and then it will prepare all of those specifications and send it to the Kube cluster so that it gets launched there," says Berner. "And it also provides a bit of extra monitoring and better tooling that’s designed specifically for these machine learning projects."

+ The impact that Kubernetes has had at OpenAI is impressive. With Kubernetes, the frameworks and tooling, including the autoscaler, in place, launching experiments takes far less time. "One of our researchers who is working on a new distributed training system has been able to get his experiment running in two or three days," says Berner. "In a week or two he scaled it out to hundreds of GPUs. Previously, that would have easily been a couple of months of work."

+ Plus, the flexibility they now have to use their on-prem Kubernetes cluster when appropriate is "lowering costs and providing us access to hardware that we wouldn’t necessarily have access to in the cloud," he says. "As long as the utilization is high, the costs are much lower in our data center. To an extent, you can also customize your hardware to exactly what you need." + + +
+ +
+
+ "Research teams can now take advantage of the frameworks we’ve built on top of Kubernetes, which make it easy to launch experiments, scale them by 10x or 50x, and take little effort to manage."
— CHRISTOPHER BERNER, HEAD OF INFRASTRUCTURE FOR OPENAI +
+
+ +
+ + OpenAI is also benefiting from other technologies in the CNCF cloud-native ecosystem. gRPC is used by many of its systems for communications between different services, and Prometheus is in place "as a debugging tool if things go wrong," says Berner. "We actually haven’t had any real problems in our Kubernetes clusters recently, so I don’t think anyone has looked at our Prometheus monitoring in a while. If something breaks, it will be there."

+ One of the things Berner continues to focus on is Kubernetes’ ability to scale, which is essential to deep learning experiments. OpenAI has been able to push one of its Kubernetes clusters on Azure up to more than 2,500 nodes. "I think we’ll probably hit the 5,000-machine number that Kubernetes has been tested at before too long," says Berner, adding, "We’re definitely hiring if you’re excited about working on these things!" +
+ +
diff --git a/content/ko/case-studies/openAI/openai_featured.png b/content/ko/case-studies/openAI/openai_featured.png new file mode 100644 index 0000000000000000000000000000000000000000..b2b667c0bb13d09866a2bcdcf39c77f24a55c29d GIT binary patch literal 12132 zcmaKSWmH_vvM9k4g1h_R?(TyJch|u!xD(tpIKe#(?oM#m;O-vW?U8fNci;PQ-kr5( zcJ1C}U0pR@UA4m%7v8^qpCI3VA$2lN-Y>Y$;+oDX_GZp* zMvfo|5mS3(5Q(&{kvT{iWMt~`bqK@{0RiP`sjBI$Dfflf#NL+C=r0bVyRE}JH3S5| zkh_DCi8aWX#2935X(vE-*4jx%VreQsrokb{EaxBwvapo$bOfn*%Bz}qTAT2gk_ict z@VoQAGq43Y8!o|bL!pY3e!9q{M%FM#W#LU6O%*nva!pqFf%gRFXpAXr)G)Gf2US*)fe`LMi z36NPhJ3H_)F}b<9F}kra+B=#vvGDNlFfp?-v9dC}V=y>**f|@yGuSzi|HA(a6}|#aV#tUDJP0!PY@e?td8DIsL~_?yh z&eA{uviB#9rk19>>?S7M@8&aNVCLjxWnkwqWoIyA=4O32ATuk7)!3ND7{v1rpZ`V= zaEWjMxkQ;o*~GZG*f=Eqp_R6CayGIv0sSM}@?G|SXj%VP zT3#_nkdd>!qpH2V%|BkCXkqVc?_^=`Kq4mgchxx zbOo78INIBi{L9w7mj6YMcg4kcL^#;FIN8|#`yBpFYx>{p_itK`|1T}myE06F9p(RV zoPUPiz40&jKVk5G^FM(EvU`sn$M-M^Li@k~0YMig4HQvzUpmqIsH;Bv)VaU%z}V2# z=U=h`i6*YJLMJyCpKjJO{bhN^0F->H*Dz8;P|CyBhe%68EA|0506K3&xWDy`_swZt z17k7urg81*=H=RByZvs(YvLtq-=kwftIMttomTbP59+5{jwnUmXG~gDwH5iC-?SF| zV6)?FEtoW#MXdZqv)WpMva88Pn|XfbrpS!Y3~eD=)Hi*SAo=3m{xgAO*Avy->KQ-* zb=WbYbWoH~9^n4&0B3TfQ^C~X>^zHItc=4I9pa~zUDxMy@Ot}|%I~FiJc$V<0uoaZ zERa_BGdYQ=w6G{1L@l&7O^zJU@LPPQgq<`;>guFYR>ZeSWAF2WA?o@>~AJ zr2mlB+eQwtmrXwuRM4B!lt2n+N_`u1vPM;-@V5EV<}EObr`{*iQ-xEW-F6QedSFc; z83~?tn`>JmXRrEIt6g^s10f8T4RAz|9aN-@i!yyihlZ$SAs2iDI^V7#K0%%x2pWxN zjxDpRlxb5K`-Cle9DGssml#L~?_L>rREc02pFO{{(z%tdS8gLfKuU+GXW)L2&gWug z4_vftTmqY+!{NqkM_kFkvzkoDf=0b|U%MvltZd6Fnq3!cZ{Gy&J&io8$NFQlIGkVU zkbgT>(T`XQ8nj-7o`zAmfaQS!V^&d3bX((E`Oi>%tn?~rAINOfmrHlRC9_^=x0FuW zcqk=#EBtiZhCqHL;?;cBZccJKU(u< zk3Nu8mk=?toUDnnrSo|^pZ1X_rX;g``Oj(?>@$RzVWn~A7*B!A7fSfbQ~66HncKai=Ur`@UW={NM=4o}A$dUUacBFuo1fMeGrQHO{qWTUuMa z_e{~xvimMMOE9O)sjbY!{1ZQDjbG_8eX*lgLeOLFb@tXKdR676KZ;{w}-9;?>a+} zd>`U|xu790M!K>*^g&PQaDSa1VDeUT-_k8@KB>oKT18!4PBI9RjqUg)vi79l<4M{Y zH0yx1W|d@vl0gAPXBBaBuGimg%zij{84xn}h>n?AI=Q|5<$GqY%XxKS=~6vu)$qAZ zabng-hXbF(B`mt}kXvu`)?Kn+Np8J~2o0H{QmH7qmS>9tK|5|4;F*j~RmE#wO(fFAUYHiQJF%NNSit5qD zZeZCxAdvu<{CwT+?fw5aY6q(Zh z-{z`dwGzntZ+qhNWPlo9$IPeS3TR|d;th%$>YdHLc^U%?W)HHmU$iZyOAmkGqUFpG z`qpNrEt0b{3(#z*#QMqcxYP1bmU{ua0t#d6EZ2W}{z6${VQzBkBw9#1TXx=X6e%s9 zTXkN9&e`ghC+T)UG$sTl0uDPLyHX+;;{gm(qOIfg`!c2*?2#o!U?>cwA3h>m z!Q;7v5)(Kq7GM0RlDw|rv~RcRLi4ur)z9Q&M|@4zo&u>tVzYR=XNn&oG$V?N##_Da zcXIu}YaR!8_cg{JjQL~-Yp7vVY2A{-xB%R*W#RZ)*eE1GK*Qlei}b#6fbrQ%b4s>f zC6b#E`>&aS{G&>Q5rhmBT)e5{MtiRPA`F{HXRqtz9yaV%HbZBjpAu{M*?h<0xB`zb zFSlcJCr$hjbd_o%JJV)wkEt^doOW#eJlqttlDwHpG(^zhv)S>;g^jg~_{(t>aDbx&X@X zn~0g{eA+>NN_h&dV;rZ`zI!dwAl7z0kL{U_vEN@eTk*+1Rp%S9nu#%MHq&0-4Lg{b z_L^*R0IO4{^H)(6Eme+b-?uei*3YeK%cvjD?00yYJw}>O{80G`C;E}^+cfTKn3U=v z0zk^VZ@$lz5~RVGvmRH)<>v`H>FM##c*x>WXZHK8lHK+!{_C(7AU&eV`8|5* z#^`(G7#`SjiWHq3S0M6jkQH3IqI(MMrqHW{+7v~X<0zP_*sI}d@wtw|kl)J`7hMI8HPKvK`B^vDK7-za z@y0ijlTsS#BI=_r>b`-y*ru5v5|!4%Ug+|kf`-!-^sUlswut2{JfqOL=v+!cIkB;0NLcil|vVv6iOJe4Pz1A=&bUOV?Z zCOQvl2C7I24)+DQn>|l|)^J{)A~Jcb9X~sM4wZ``VCNOh-~KVMK;Mumgmkt{km+-R z(8G*TfC6~~cZ^r1V+(MdiWp0L-xpC>6-*81qYxA6$>>F|wHpPjkthkI2n0cciJ_Fg z36OKy4k%as%-4UBu6Mqe3};1w?|%qb-wB*g`>Y1EniiQ>-W$#y2I=aio6{j9EA%wa zU>l)_GXD@v3XRUACwH?)P714l1mjP~n$@y1j3h|1G11w#I#vTGnlC6WLnb(j2?Tkp z9;-;V$qC8P?1am184MmC3&DveHwL!S0tpT7x61A6_8f~8N_JC2ZhvH*{@@UJ3szdJ zg;8B=i*27tq7_E<5+vl7Q(C9w$iMpvOdbi*3i?DmUS#3(O%gmWjB8YM#B!6dmy$$v z@+@XS=F|0}{Z?Mg%G+rBa1f>9dp?M_0(ws$sEk$W>ej64o*q4-(^wpPf9GnqhbXpu z0*NQH_x0%0wkk zTTN2t@a&V%qJkSDFUw!Q87Ws9%0?lIM6}{FKEuZ6Y;C~Cbdb6Whv5E6VcD&c&;4*E z3GS+Y?SY)`))%YvTc&#L=ItV=6x1U-9>xJ4ypOT3c>uJk!4(E!v_L+{C=`-Y=F z4-9p73=wi0^T_ImiyNZj>TFkB)SB00ZLbd%V2IaaEhtUlOXdDX!WJE8M!U;XvP*D= zWIlSQ*Hh^9vvSN#B&(9qPaIB{uG%vJB@PibzGu`aN+|GGO^f z1zoVFk`7*SzCLUL4k?rUe4~MzDR7{p{A26QA=XN#hTcc7#`B-Y#iNO%H|ushEnJH^ zQ*7XIo0>S-s{1mkk1>A?ZkYYCJu-yjQOKyZOk*Zd=vlSOeOI*wZ|1s@{7>QLLu8BX zJwxw4;jghV2!=Anclwy@)({(1ggS62c{bY|Pb}ZP;F)km+Gz)_G7i>pziy(kVav@Q zu+Jim%kzvapBCtolLJV<$XFNu@x5CiWO43BX_wuzR-N~TinxqS5xn#-Z2~6xd%>?J zoMF$OSXaRLV{^XM*$z}Hf%>o*^1nk5*>Ke#?7ZA6$Y#`oeP$xC&Ds*aD+^mm{j$U7 z7HoSf@fCP-cBAZ4D*>uj*>C%g2|fEZm;6B4G0SYDToYn6c7DU3izrV|N9|3#|DNHu zlI6**QoIWYsyaEhJAPdWRe%n0E~9hN^enZJ-|5lrH>}Yi*16nB^^$11)b&gXb1{B8 z%Jc&-skyFwLN%&Ot z&nrIu`N3jur5EYSdvUnMglxk*Slz-Y!jR(g##+4ln^z-TlTSIe$meM-c3*|bN;4V) zpv#b&$Syk#u5qQ0c>}u=4GQ*H*N9eUs^xu-=p{ym*QM3=|atizE3f=}k{kQzWYu z(=rB_rL=|>HLv^0<8N-^M{3%(+Or#d>cx%=MIbz%+?ie>w(iH}QgFW4UG{Kbieo+VB@cD@3bJ9z9Seb(irYmNb(>XN{A3YgD5EvL;K@Tl zZ_LhJTj$X*HKfHcRwhXNUQXNyh5!|dX5GWsZ@pfUU) zQ|aro-FYwJiClEpM$7dj(8u2;z~_BMb>*yN_hI)w3zkib>AWc`oaDoWa-UE}U+qBk za^j>g+E>swE6ftb@DkBr8Q|YS*1zEY6t&(xH}`LW>m5(^FN~%X{4bd7r{Z7uFTpF> z>#cYyQ2)IOFf2IG(+ZP& z?#mnp$f1dT1nWDdzK#zr`zx^55n*hykq?#S4ap{4K1!a)g&ke$TM&Q@1VQ5eEh=#! z4lbTVDcgutj0zC}3AsZpc1M|dn90}TKA$(bAt9`4)NB#uE-+NVgv**I+koBe9x_lK z83nkpG`(IjY5-y_z$olHH_`(t(ovOw~B#%a$L? zWn5h1=VB>ZyZ!h8HgkNfPj(LyWs6qT%V71$OQqLvPXGACRo_ZH_-96O(FYNcT_vo@ zRl8Qgz6fe5p^mqa!kHAFaMsKa;h>}Gma>gfm%)F(6Qt&=(|0lpdi|cRJhM97@25Z z(-WO~CZ8>L=N)gVG2_5Ld=L{0ePzZ|+E|Yf69R_c1_4YP{iZHg4QITekZ{EdhybL=aq_JZ^CbZar3yzNNIR|9_T-7 zK;iG2BNECm?k6R zN2pnT|LEz!S@+0%?GmwzCj%uHpd#d30?AXu&DXnUd@g||P{pEI^?*d*aUCqkOA{y( zUbiqN+o7YI89WPV^yz(Puv-d{R<8|C#-U#UDmNu+)L6ghCt2NWyW2RbD|0reON9{R zr}4aHd1a?El2ANSQuEl|#do)LXE5sai{v0+s)f6-$B}#>EXzZEgCrvg7fkoyho|{k zS+mU?!%gIIlbd|1&AGSom2Kb_mx%1V?aFj84wpvj{T#lki+Jfuv|SlVMDQ~rAWXLB ztv%SNEa)r_8(KeZHv{>$8Y_I=MRjB&4t9yc>;6D+p|QbYb)_}p2#^Gn6eBVxp6R>1F=Zg z7`4R6y!ci}n$q&s{p#Y2ER!i!kP2fu&&E4zUR8GOY~O-h+c0~rPPK2 zb?~AVrWW5HWgSwdIdg9MgqXyXJtCVZoo22ul{GxX-m9r9-}NV}J)ys1$Y|Om%TcLs zrYPyU8?9>KBijfT7-_-Qc*-zsHc3N2%@apx*Yh#cGhbG`iculho%=Gw+}TA4wj8@J z#XCpzAAQdwkyAxO2!c!c{;xZis^y2l$u>HH~CmL?E?#zMCA@+I^5+*}9qUdLVzQ20qOpP`)g|1{OX=;68icE^oo9%e1 zZlFkIFSLwoleX7Arjhb{H{hAfSKQ8OO^j((Vaf%MZS=BI|K)WsD<7|1qLOU`MMRmu zRxbhcNjkk3W11Bq-j{v*;C0))_H(fSZ}Z;K;(fynufTxS57!`!bCX{yD&`%&FQM?} zj8-l`YE|G+$@7G707#H#rU=?5W^$U0L`*z<3vdrYh2d5 zVn-IM()mFwXlll0c?GNf>Cb(&sDS@l-R zkY(LEqYleih4{WaW-Y+F@x0-4r&&z)akJuT(BV z1Ub+2i*=>p%#A*+^~BK**N&Z2q)HolEh2lmnxLlVU{JUS_veP`revhh0A(`^=FR?L zK+x*I*DHt^j=KIezP%-dCflIs1Ct~Np19nD>mEz9{09S(UL6zEYS{n{p6J!08vR1` z^&Rm=Z*9+!l8A1UBx4R2oAAK(55goy>nsv`;F>TBhw>EoEIUUvMwLe3( zwz`vmrs2*>FF2}qG;31;4_uPiq#819xDO9V*j$(l+rUp;kD3Z?q;y~p&H0g%<@ao; zypsu~7aS(lT>+tXn@C4E%vH@>C=tt%3)J8_+V7#%8=#hG`00)z`$p(`kWQA)NCht5NC+Ybmv)DUOhTZT6H=Z4n^KC zQH58e9E;q;d$K>_D<_mHA76k53(-{_a$p;Z4dT`=0OX71m zKKNr42M+l1AtAqN4Ue5OuhjTp{U|*i?Rrq^uo^n4)NS%w6yw5cwv+B9)lm-)N%at# zBI|&Sb|*fCsARai%ucC?|NaF0wN@g6Q}+{%NhIP>LF$f*H12H7O0zFlu@P#(f}D2w zvp}qTX@9r1!xbz4?Gk^+Y|*tRDz(t5u9>5mPqQ)PrMmC*N^6K5k!N)9xMGdVfO>2v zOQ(qxm3g@lR{;6a0bwZKbpcHAME@Btk`#p!WrTO7h(8DV??5V~ZliCPgvyex6sdId z6=>vh6*`qiQ4TlkB78wC2 z+!A8LQ@$uUO7CsMqvi@fMXHeeOj2mE>SjyxqwkvuI6TdM2RR$sezUlBCVz14>K7|G zv;k-ujLCBbVjthaW|`^Dn0h&w!jNZ@8V-U_q@bE5dkEg>lAf6Ij(s!sn^3Xg5>s-y1!GI1Tv}N_tcgC@l`_xg$i5U)@OMHOW6#<8MvGZ?^m3DLU}#d!$RAO%yVmfg zbWpsD1C1R~2w2Sp)4-jwY6OzRKZPwyi3U+nK<}~zw&l|~IH&&EC7T<&$GAZh5e!nY z57GoeaX&J?79HLS!>XN#e(z6u=?up-4;`*g>>X<{H;0>7sNm{VFPThbxmfGaA2xo2 z;Q8*h@JG$%aiO1kusn<$NE=y;jjfx5^K-5XVjUR?MZ6ZmpBw{rnAu4Lg~Mf?T|+e$ zXI240>&*r8Tm)-N%ZOhCtb~V}3>BOEBlD zy|YGcvy5SH6144+Qlc?!9DAT>6iAMX^4wu_5k!drxT$cztl{~MWvD^M^Gr`?aPd6; z+s@vhS%-1I+n4x_dw&xTt%poVKK9q~9okxru+GlThdC6bLm!Q4u6a?3=4<7}df#G+ zEnQzy@dWuzFmwh+N$4<=`O@G$rdrx()y%MY&)XzB+q7=EK?@Ej)$a-$A@lw}et$5_ zkp5W&rHC0Df@kFZgP|}NE4bW!dJ`iV6cn5Pwg86OL zMkS<`))s#Xq5bXdfD#3j$Nu;+F0|8YIZsCnxN;8wKn{>Fe<4Ft)Ld~nSn7}t1`rE; zuq?Jy3~`;oLmRNVX%m~|&fbipJn&qJ{>hV3wo9ERD;?3q<~=&9grv1ZeHKnpn^Byb zHRo8i_PuoFM-`RGz!8Xq4?v0m&RYeLm{*+~$aQ)5&YIIT@Tzh*1T7fy251RSvbUSK z{2qn5JFxS`&+~&q&1k$l#lv%8tX-JGph!1nO<)~RJA|@Dfn?e%&MYLM;rqeISqasD zOO79A8MoTcrCC!wDkZ8ae%t+XE_4@XC$38-JDA$5{@P-_p784vDGg_EqV&hzoZ3Zd z8Y-Cw*orHoh!V#)UzVYwf*|**6=yADama4!(_@;owi$(S{PwtoD z;OYL9IJ6XPZC!cDgA(pVv&Ikvgo<@H;ZIJ}PI&Y@$2*qkrUcjhjg%(&IxhUFp|Pz& zCffU&Fc5UZUm7=MyIt=Lf0JtW48}AMBm$Sf{79eOpIIV@2Od@di(|tB7vfc#qr(lx zUAZqPznHIo5r$@TAxP$lz$R7a(Z9jdll321lDmSJ^38DuPBZ~ayLVYn?+Lzvb(At) zvjga1#J~w}eA5j=9dWfMQ1mF>WaGL<*o^(!|2?$Rm{Tj319iS6t4Ol3pb~1!B%mCy z0qTwzJh<*CX>|8%!rsIb#hbCkMZmnbC$7RG@MN z?j@Yp*2`eglyQCimnq(p9EbV@gv_)AJ#FQ^#lZza)uNC$@k9Km2Et{83^d%%8=N8G zQK|@1r-TWxUWadgx8;q~l&X>M{!pxYzSP&QQV9l5>z$@gE&fKc5OF|x$Jdy8B^muj zB%r;3Gs{~|NFPQDby|(gURNBW8uCV+je}|IHF+8e?G>ARZ!rj9e*5WIeTgszTgas1X2?ia|hgb1GA zXhrDN?p6dGbp1|M2TnDH;?;WBw@tmT^+YW`a=*{$+-eA@XmEuJC6--mXWY&aWl0RV z2zi+o?5gPwaYXm(Mk@{M>a=WJB!3jeW<8xGd9AsdPkjt--anH0rPnzhLc36U{?p=v z;tUgUq?AZIHfH<~8Gm>IyF>j=1k;~CI3t95)kmHlktsvxTQeW}tMM zT7-vtip`7|sY}#OVl)Gjpn|G3Llx3b{^7Ek zNfiChCgop>;Cv-vU9Wo=Q%W)zw1=N=IvT)^5c*?#zlsAO(jC*gASbC*==w|Og7S5<@vK!YJdDNl zu>^}Dqw>|=$7tMx{<@0KU$KZMU7{&?sCO2wSZZjJJG7Iw=OO%aS?PRN7gY+wxy{~U z)q_JDZ)D%)^+2S%|Jz2h0^@$tF>WB6i@#>@v5fhTg#3Pe*d-g$eL=#khx4xu{q*Qm zysoyL4vTF}Dc&lnIzTphaj}rC+ZR`6t5ZR91!QR@b}>g-to3Rev5w3(Dt>qK?7_Il zE89=(iPP#Ow0!I$r=1yeqh9m97WLY})es9FPv4AQ(jO(3+-#jJoDjq&@1?nuqnCY5 z2k%AZ=kxYWorX}oKM){)T5_hM3>|j*78PYuE0iKr4 z9Zo`>2`|u}M<%Yq{;ckL1gZT6T>y`S1AN{jYggk$&Lp3CY=uxtq|U7qu~5CqY{e;^ z+ot2Eh~D=m<3gE$dsCkjYYi&a8mIeZSL=cu#3Bd=*0F(YZTR9-Ygc%X~eiXi-<)>szs%-_rMVBvC zm-l)yF|X7V#RJ_NzuLW*@2{iV@~tNz63FbgXluyz59eA(;E9DW2{FGX$9;VkW;JqA z3=nN6_j+$@b2@&3R=K@3er?qE0HsoMJXKyK>y4vZVk>aXz~rQq3C51$Ky7z-*P&iL6X860wStlGZTT|}hZru*)w;-W3% z3z*ZkNx>Y39*)IN^|#gn`+bWwdaeLPP+4Dy$+dN-l}malJe8FTulaZ$k&+_^oht6D z_Sjsq@0fksoSI{npO8Z>x;R`TDfq+JLKPf@=%J4p(}9-sY=X?^N`z%9dx)F!Hg$}# zk2!?~k+iUI;oLYoFrX3=Io{z;>GEO(+c<~T_{`f6DS@(2txr07xNnMkSgky;ehZy$ za+*2x=fYE?4DQs0^}-Q~^Hnd!IgI+{|GcGI0&Mf3ribv}k&RRyFco oBbQYHlkUGZ2Ffi_@% literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/openAI/openai_logo.png b/content/ko/case-studies/openAI/openai_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..a85a81ea063d0cd80218f48c376547313a47fd73 GIT binary patch literal 19818 zcmeI3c{r5&`^O(DjV&`0N@`4$gfW(pu`{-V2+39qGuB}+V<)l>Av&cfYf%o#l6?(P z6qQ|;QYeW^h%CQRoto3>ocVr#*Z0qvYs}m}pZEK|pXc>i=DD8hdZG=FXtA^GVgUev zT}K;jMEUNg`~`y;DW8m%m!~OT%8eMl@UjZ*&(DPp|Wz~2pKua!_uNqNeKiJE+Gv^$cZ7OP?Wc{1oYbrro=*t6g;u^ zC?mAyw{n#K6k(1ek~<0x_wn%&_mL7Ocsfw*a&mA9Nw}n>7)2pQ^m8L&e8t>|`@Sam zGY=X^wDWXwCpi(^pc{EHwgfMdA`G@s==%+4QmI-;c>1wHxiNJL;TX`?nodJh>nDRDEYnn9}82)L|^}V?Oz_RtLraI z6G^NVif3R)z ze6@X<3SUP^fpV@Wbx$0IMDR2r5b#Q0XPM!j2%+lg8?#DATm~wvkFj%d+aP;T0sgb? z$C&@fh(lvYIHip_DJCIDd7B`RC`oCQjI5Z1EJ{LxD#=&nPdW4nSSNeGU*wQRpb#?D zIjEFB=b)@1EQW;nkFro}sq$d$Q1%2*R}4wX$ra;(gS)#qD8PR-QYpXBER;F{Pw=E1 z7EVb@0sgCID%Y3UhSG5(k}z&|I32VSr9s@u35&9qLrUA>q$$f>M)I(jG!lyw!ysjF zVhAJ#Cu@KBu&uN$?h8bHeQNvPQlklWUK=asTWaionVQP}x6}rnPL$magQwDL9Q}V= z;umTDR!ZB6NSUL4RCAXygTL%&#yF3^w*J|Hclxp~xnn$uxQ$h(2>Wx{ec#Bw?^7F| zFRK@YvD?_ul`6WtPn@a)WlsF3>-}ryzgmATp#b0b6~H_FbuhlR{&kH0oV9-+Q-zJq4Mmv`lw-qu z83%*{{BLbPDu30RZHz14>x*7iQiAe=BQ`$2S*g2!wwnLDmAd<9>ksEKHz$&k7fnRe0y-{g8Z=`%E}Dp_ z1$120G-$?jTr?3;3+TA0Y0!-6xM(7x7SM4~)1VpCanVFXEuiC~ra?2N9}Yjq889`QPZFq({a&6L@l7>qNYJJrsJZCh+06$MNNZdOvgnN z5w(Dhi<$<_n2w7kB5DC07c~u<@xS6?`SB1S&W-ZqpAY2`zuBzza?0a@P&;iSeE{$m z0)UV(09ab5e7^<&F9`scvH<|pB>>@JWI%riB-@)k~W&A^f?uYZ&SK>@QoVzZlAtI1)4Ro;? zv0oL*6ppiOPQv1c@xwAwDUYoJaz0%!5`&&>|=i8?)L2oSI z?8e+Q-4rqA9rlq^_&k{(pf3-5eJJE+vz>lSDR+r`>{%D~$@hJ3qxgUmICVv$ZBT{r<_@bz;JK zjP3NIgk-e)E>K&pB5etgDIM%}NHTVcDP+-Fz;~0WloG zpPpUi7~vVH4vjBu<`14+DcbJ9BXVrEMg|>uuA2b>nc0EwFth3k#BX~$I^W+qGb$tb z=yY53`Wb@aaa|+8CQ1GCfx6|F2MC`BypE$4f^ieGb;02Y`I-H^00e+-{~ZVgGRV-D zL|npn&milqD}gUQW-I$G%z0pE@!UBb>kk*>t!h1_;zeQ2iCfSB`6@`zFp&WYJEVDO zym_FVaHaBNt3mMc$o=;VTx_ue5p2QEMNeWb^i0W~dS{E($Fjqi0oE9@&xoIQM?hy< zgvRF05J)Ciqf3^jZuRO_&+Ac5vn!waO9z(JxCCH~AmEJeY$~xselpbolaw(BZm`(C z^K;znI@Ygr@OeR0Iv8LGFR-!Y*>Wzu-Y|FAONZ?Ty{mEm3E)zu*RNG28y zU2SNBe|?2XrBeB&oLm|1aOOmCe2w>1d-~d^x7wSm_9JVYwaQNB1|0wK-0#b zr4>0SFGd?7kZ66UEj5eBb0+JTn$oj#51>_461Go@uP>vgk8m?K+$UaC2|cW8!}aV| z@KSX2V^x~=y2dgvaA}`d zUeemrFRR39-ZilTIhqSro@?HR=IW0Cxs+lteqJZu9RR@j)+R!>!?(QU;^sFvgzYM;S*B>BLwZti?n&D& zSYe?A5A3O>2tJ3zu)kGPp#;hj5+Zu67v+eCbX>^dgrv;JIWVbge)m8=>T~(vNq<%k z<3BRv$Di{9f*&3*s%VF5N5**6EFEWRs~wX|h)quQdGkh61ttiFV8l!&i`*?`_@BVB>+C|MYZL%+z$?ObFMQ zu>SBpmC?FretD+lfB@tfM0;LO@=`YIaR*Zo7sne(DByN@ME|ZVMR~3L-fZmiK8{C* zE?J{|{8&`B&Ftah+sT!9NNSEy?~|P**wdVd&~S0?nBBWKwI3lZX&9fi&7ro+wEf8q z5cy-X$2pV>@`Y__61qr31-$=STa}Oq3?`~g?iPX+S&}PEb&0_{?D)h)k0VD%z&oD&|t%EEPzPb=2@;(f} z;(aJzsKA)1RbjWYvs)RVCJH=&1_X+|^T?oH!zMCtBRj5L|GAF}_`UV|=L>{-cj=gX zq2fq22hrk4Hgokvgigv@SMb{O$Gf@n0e<5z2WIPit$;ByTUHS^uBe6A1&_6LVLQ6p zT;;I$*tE~7!2F<6v&~>NE3@3dH%sa3W6c9`8XRsFq=;aj%WX)-Fao3T#EQ4-!M2$>O zL$8UuF5U;oui8FZrJ%s>Y3gIL=-kO}KL#`!Cer-a#~L*ec*f!n6BlKLWzAeLA4~-s z((N~^7z)jhxSlo;09I{K+ntK?8DEZzNa}%s7#U%}?8$1?NHrozirZ%Du$PiV`>rOK z)=BxEQ#hlop-?pdX6yI{yURQ*IY=EC>({l+R>Nd^pYiF^$d%K!XHbQLiykK(+Fa5g zEJ_v@(}y?@w(!14<~TC_X@#JCDQ6E@C1c3=N{Z#DJMFE%cMFgON{n(x%*@=*ypl|{wl+v*^YxM1VwYB%H~wIBSfiR? zWBRFAkF=Bu>x+}148Zw-m;)VFzOO&YqXB4Q=5`0T^3tp46NqhhYRYz?-xae@^xtD) z48!V+V8ExZ)RDnYI;wDaXf{QI`)0zR+o^_eymh>$^~9l?i7+-!FlI)q#X4ZB=te29 zYv;6OUiN6f>mK1^;ZUHfWbPrl1}U$FKK;gNDt-0IfyKD9Ktufgor|S#rG<;5fhR*C z4d4@LDXvv7LO>y0y_a;m&hn$)Rch_Q%SU$xuAI)3o#;H7BH03HPYrqETzxZcJ?HRc zp4)SWm+jiCj&u{SZbie{?vWYFU@PU}=;2|qC=qP)FBcCw~uV;i!tEps+iqf9bCnV<{$)L|q zH!?7cExarNL9=X5Rc8w|Afuu(Jx7@^yjg*1Z2?6OnIRb#`hthVgpe(qUXn}hzJ>5J z@?@QJroUesn88=*lf(FSGeLH_sO`UzU10XAo2R2U35XFGd}4HCD>gwBeyc|-9RJ6W z>jaDCtsOFl>Rpp9&0heU83xN*HCsa%mXXdHWHXm8!7YMlv%Ktw$|_vbf}E#~auW^{ z`t+Z{R0sn0V2EaO3ejmSqx+hy3Xm9)?02;Z%Hp<%kx`(`NG=v1{c>F6=7gL0u^S_q z?H5!|`a`oGz{R#6i!1FxxBo!te>SU{)as5+dagAg_w3&gdV^ZB)n=2mxu8 z5!u~tBGAk*U{lsy9i$Cylzt2e#m5cYt3C5V`_kh<364w1RS;ta0E2RZ2+}$y$@}iL zxd7pU?^9nWhX_b1mVfG7|G>xqvN>_F+u*QC+ekohTxHZe6FXxVDC_=09vVM$KKva^Ke9lm$qsZ{PEiFd3S8Bl2C?On@3A!HR`;D)}0KZtS8lNFZdIhf497Fv+3 zHN6XF>wV&b`Qpvwd%K-+_dh-5DgyICY_~H>Z#@^cu=7-9QdQq5YH!;?m2)kQxs(e= z!TjiF0p+LUS(7irMYtFwD}_VmLkrvhrmCpuXDzq!!ue7MVWPHV6=aUjYGAhIG~ZK2 zg$F&`So|kfkaeN#UA!4#k?rEekM{Ro*4~cGPF6j4ck&&?+?MHD%aJ#C1o=qY?v$D* z=)40gxV0u>$0cI_V99WIsbcd>2Q!2yo;|xw>HWP;Bdm}u?3^N;3I0W)#~tjg*T$VP zuDHMP787VQdX-E086oa2#^`G{bXU7p$i^jwWk|cUUon1;gy5gVH$eL;;|}@pi0?HB zf$mM@gJ>S^^f;$6p3nN^zWTjJOjTq=eRiNz3O7+%#fBp$d@fO8RX5O8Y1A9n%THq#bYzU-cnRc*nD|>#Q=~6={Fe!12<6Hs_jTq_>PRYWvFNo3&N=Ws4MB7fWvT-=y?PB7NBL zABY`Gh4GTyS)?}QrF-Q$6Qkz?geVuzeu)9Nga4bGygN z<%QWQA(%z}na|xzYjtT4%YEWqbkwvj35m~$aSz(K&BU;SdL!5)fhOC^G zc0pqEY+~illqZ$U7Vgfs@qV~oBrPK=p@e#NNnLaQHOht6!*T;7YyICR0(Q!FKKZSd z1yISYB<&Nsa9@i7;M45p zO!JV$v|g8J{#e;2j}nQuj*p@F<-8Zjlrqf^9N9uLPUlI@T?ydkFDf+A;pEZK3{S=t z4n5^0?NVi8_P`VspWY+UV`0n20OBlr8*eV`yH~WQ+h_MKhs5BuTUU7sI(Df)7%~=F zS=y_3LZsO>`WW_c=f^eh$(_rL6@{m?5Xop~!nR065+$p3Q~ibd+Zd(d!LEyJ0-WA>SkB2+}1#Z374 zW;&}zHang34*V^_JSnN~x9kdDt%&~kZAq5)?M>Qsc#}x!9iJ+fgkgH+irdc*kxlccxGCmR$xX-C`a6p;0 zq0G0de=`JiZQPK98RptUs`_js1yl0=R2S?!bKNftt@3O%cv)8+d)b~FeJ$y*G?#g( zRjsAVta8Qm@xEgDJD|-7Qbs zNf7Q-1v>JiqSxAkySew@V_f!edN6hoS&d8P5=)de3YZXnv+5FNfbF4IMJ;+Eb++~U?mu-<|{@`liB^~JrS<#jKy_#45VfiItQ z;|-&FFJ-q)-6!S^%=Mv}9`D*d7?LIwKc@bD%BY9m!Fyn^5uwJWJ|baiB!IW zDB&1=Cz`OlP+9!P+=FSw5|y!`;T@3jxrIA*M$Zo!DEYr%u62N0EBpBUQEZymW5fiW zh>Z@9EHoi&F@2sHu71w#^fJ~p+9 za+NM5e+=UG9+4!6=uEX}GF?{Y=NH*~WoW0U=B(dn!O=mH0WY$)TxX8ftYU>>qI|?4z4=-$c%UwP&N4bK`wjg#$g$cy6 z9<4Nab!dlE*W`mvtD3d9dsksKEg4CI!SAZijt1NKhfBpC|P!$m*eAnbK>x|`PM6+?}ILbg`ctk*xrKNR1xkMY*Kz-AB2n7F=SI zuPpLJ=CzhbU-m?x_)Z2DX{ZQ +

CASE STUDY:
Infrastructure for a Growing EdTech Startup

+ + + +
+ Company  Pear Deck     Location  Iowa City, Iowa     Industry  Educational Software +
+ +
+ +
+
+
+

Challenge

+ The three-year-old startup provides a web app for teachers to interact with their students in the classroom. The JavaScript app was built on Google’s web app development platform Firebase, using Heroku. As the user base steadily grew, so did the development team. "We outgrew Heroku when we started wanting to have multiple services, and the deploying story got pretty horrendous. We were frustrated that we couldn’t have the developers quickly stage a version," says CEO Riley Eynon-Lynch. "Tracing and monitoring became basically impossible." On top of that, many of Pear Deck’s customers are behind government firewalls and connect through Firebase, not Pear Deck’s servers, making troubleshooting even more difficult. +
+ +
+

Solution

+ In 2016, the company began moving their code from Heroku to Docker containers running on Google Kubernetes Engine, orchestrated by Kubernetes and monitored with Prometheus. +
+
+

Impact

+ The new cloud native stack immediately improved the development workflow, speeding up deployments. Prometheus gave Pear Deck "a lot of confidence, knowing that people are still logging into the app and using it all the time," says Eynon-Lynch. "The biggest impact is being able to work as a team on the configuration in git in a pull request, and the biggest confidence comes from the solidity of the abstractions and the trust that we have in Kubernetes actually making our yaml files a reality." +
+
+
+ +
+
+ "We didn’t even realize how stressed out we were about our lack of insight into what was happening with the app. I’m really excited and have more and more confidence in the actual state of our application for our actual users, and not just what the CPU graphs are saying, because of Prometheus and Kubernetes."

– RILEY EYNON-LYNCH, CEO OF PEAR DECK +
+
+ +
+
+

With the speed befitting a startup, Pear Deck delivered its first prototype to customers within three months of incorporating.

+As a former high school math teacher, CEO Riley Eynon-Lynch felt an urgency to provide a tech solution to classes where instructors struggle to interact with every student in a short amount of time. "Pear Deck is an app that students can use to interact with the teacher all at once," he says. "When the teacher asks a question, instead of just the kid at the front of the room answering again, everybody can answer every single question. It’s a huge fundamental shift in the messaging to the students about how much we care about them and how much they are a part of the classroom."

+Eynon-Lynch and his partners quickly built a JavaScript web app on Google’s web app development platform Firebase, and launched the minimum viable product [MVP] on Heroku "because it was fast and easy," he says. "We made everything as easy as we could." +

+But once it launched, the user base began growing steadily at a rate of 30 percent a month. "Our Heroku bill was getting totally insane," Eynon-Lynch says. But even more crucially, as the company hired more developers to keep pace, "we outgrew Heroku. We wanted to have multiple services and the deploying story got pretty horrendous. We were frustrated that we couldn’t have the developers quickly stage a version. Tracing and monitoring became basically impossible." +

+On top of that, many of Pear Deck’s customers are behind government firewalls and connect through Firebase, not Pear Deck’s servers, making troubleshooting even more difficult. +

+The team began looking around for another solution, and finally decided in early 2016 to start moving the app from Heroku to Docker containers running on Google Kubernetes Engine, orchestrated by Kubernetes and monitored with Prometheus. +
+
+ +
+
+ "When it became clear that Google Kubernetes Engine was going to have a lot of support from Google and be a fully-managed Kubernetes platform, it seemed very obvious to us that was the way to go," says Eynon-Lynch. +
+
+ +
+
+ They had considered other options like Google’s App Engine (which they were already using for one service) and Amazon’s Elastic Compute Cloud (EC2), while experimenting with running one small service that wasn’t accessible to the Internet in Kubernetes. "When it became clear that Google Kubernetes Engine was going to have a lot of support from Google and be a fully-managed Kubernetes platform, it seemed very obvious to us that was the way to go," says Eynon-Lynch. "We didn’t really consider Terraform and the other competitors because the abstractions offered by Kubernetes just jumped off the page to us."

+ Once the team started porting its Heroku apps into Kubernetes, which was "super easy," he says, the impact was immediate. "Before, to make a new version of the app meant going to Heroku and reconfiguring 10 new services, so basically no one was willing to do it, and we never staged things," he says. "Now we can deploy our exact same configuration in lots of different clusters in 30 seconds. We have a full set up that’s always running, and then any of our developers or designers can stage new versions with one command, including their recent changes. We stage all the time now, and everyone stopped talking about how cool it is because it’s become invisible how great it is." +

+ Along with Kubernetes came Prometheus. "Until pretty recently we didn’t have any kind of visibility into aggregate server metrics or performance," says Eynon-Lynch. The team had tried to use Google Kubernetes Engine’s Stackdriver monitoring, but had problems making it work, and considered New Relic. When they started looking at Prometheus in the fall of 2016, "the fit between the abstractions in Prometheus and the way we think about how our system works, was so clear and obvious," he says.

+ The integration with Kubernetes made set-up easy. Once Helm installed Prometheus, "We started getting a graph of the health of all our Kubernetes nodes and pods immediately. I think we were pretty hooked at that point," Eynon-Lynch says. "Then we got our own custom instrumentation working in 15 minutes, and had an actively updated count of requests that we could do, rate on and get a sense of how many users are connected at a given point. And then it was another hour before we had alarms automatically showing up in our Slack channel. All that was in one afternoon. And it was an afternoon of gasping with delight, basically!" +
+
+ +
+
+ "We started getting a graph of the health of all our Kubernetes nodes and pods immediately. I think we were pretty hooked at that point," Eynon-Lynch says. "Then we got our own custom instrumentation working in 15 minutes, and had an actively updated count of requests that we could do, rate on and get a sense of how many users are connected at a given point. And then it was another hour before we had alarms automatically showing up in our Slack channel. All that was in one afternoon. And it was an afternoon of gasping with delight, basically!" +
+
+ +
+
+ With Pear Deck’s specific challenges—traffic through Firebase as well as government firewalls—Prometheus was a game-changer. "We didn’t even realize how stressed out we were about our lack of insight into what was happening with the app," Eynon-Lynch says. Before, when a customer would report that the app wasn’t working, the team had to manually investigate the problem without knowing whether customers were affected all over the world, or whether Firebase was down, and where.

+ To help solve that problem, the team wrote a script that pings Firebase from several different geographical locations, and then reports the responses to Prometheus in a histogram. "A huge impact that Prometheus had on us was just an amazing sigh of relief, of feeling like we knew what was happening," he says. "It took 45 minutes to implement [the Firebase alarm] because we knew that we had this trustworthy metrics platform in Prometheus. We weren’t going to have to figure out, ‘Where do we send these metrics? How do we aggregate the metrics? How do we understand them?’"

+ Plus, Prometheus has allowed Pear Deck to build alarms for business goals. One measures the rate of successful app loads and goes off if the day’s loads are less than 90 percent of the loads from seven days before. "We run a JavaScript app behind ridiculous firewalls and all kinds of crazy browser extensions messing with it—Chrome will push a feature that breaks some CSS that we’re using," Eynon-Lynch says. "So that gives us a lot of confidence, and we at least know that people are still logging into the app and using it all the time."

+ Now, when a customer complains, and none of the alarms have gone off, the team can feel confident that it’s not a widespread problem. "Just to be sure, we can go and double check the graphs and say, ‘Yep, there’s currently 10,000 people connected to that Firebase node. It’s definitely working. Let’s investigate your network settings, customer,’" he says. "And we can pass that back off to our support reps instead of the whole development team freaking out that Firebase is down."

+ Pear Deck is also giving back to the community, building and open-sourcing a metrics aggregator that enables end-user monitoring in Prometheus. "We can measure, for example, the time to interactive-dom on the web clients," he says. "The users all report that to our aggregator, then the aggregator reports to Prometheus. So we can set an alarm for some client side errors."

+ Most of Pear Deck’s services have now been moved onto Kubernetes. And all of the team’s new code is going on Kubernetes. "Kubernetes lets us experiment with service configurations and stage them on a staging cluster all at once, and test different scenarios and talk about them as a development team looking at code, not just talking about the steps we would eventually take as humans," says Eynon-Lynch. +
+
+ +
+
+ "A huge impact that Prometheus had on us was just an amazing sigh of relief, of feeling like we knew what was happening. It took 45 minutes to implement [the Firebase alarm] because we knew that we had this trustworthy metrics platform in Prometheus...in terms of the cloud, Kubernetes and Prometheus have so much to offer," he says. +
+
+ +
+
+ Looking ahead, the team is planning to explore autoscaling on Kubernetes. With users all over the world but mostly in the United States, there are peaks and valleys in the traffic. One service that’s still on App Engine can get as many as 10,000 requests a second during the day but far less at night. "We pay for the same servers at night, so I understand there’s autoscaling that we can be taking advantage of," he says. "Implementing it is a big worry, exposing the rest of our Kubernetes cluster to us and maybe messing that up. But it’s definitely our intention to move everything over, because now none of the developers want to work on that app anymore because it’s such a pain to deploy it." +

+They’re also eager to explore the work that Kubernetes is doing with stateful sets. "Right now all of the services we run in Kubernetes are stateless, and Google basically runs our databases for us and manages backups," Eynon-Lynch says. "But we’re interested in building our own web-socket solution that doesn’t have to be super stateful but will have maybe an hour’s worth of state on it." +

+That project will also involve Prometheus, for a dark launch of web socket connections. "We don’t know how reliable web socket connections behind all these horrible firewalls will be to our servers," he says. "We don’t know what work Firebase has done to make them more reliable. So I’m really looking forward to trying to get persistent connections with web sockets to our clients and have optional tools to understand if it’s working. That’s our next new adventure, into stateful servers." +

+As for Prometheus, Eynon-Lynch thinks the company has only gotten started. "We haven’t instrumented all our important features, especially those that depend on third parties," he says. "We have to wait for those third parties to tell us they’re down, which sometimes they don’t do for a long time. So I’m really excited and have more and more confidence in the actual state of our application for our actual users, and not just what the CPU graphs are saying, because of Prometheus and Kubernetes." +

+For a spry startup that’s continuing to grow rapidly—and yes, they’re hiring!—Pear Deck is notably satisfied with how its infrastructure has evolved in the cloud native ecosystem. "Usually I have some angsty thing where I want to get to the new, better technology," says Eynon-Lynch, "but in terms of the cloud, Kubernetes and Prometheus have so much to offer." + +

+
+
diff --git a/content/ko/case-studies/peardeck/peardeck_featured.png b/content/ko/case-studies/peardeck/peardeck_featured.png new file mode 100644 index 0000000000000000000000000000000000000000..ce87ee2d47f7b5701843b8153ed648c46444bd9a GIT binary patch literal 11786 zcmch7WmFwavn~V)?h**@ZoxG;o1h!_Y}|F@7Tj$E!QF!sf^8(YyL)gC?sj?4Ip4YW z$NkP)_s^ZRdb+1qJyq3JHPbcS9igHmi;hB!0s{ks4v>=qzSVIt`i$!@$5XSgC7+wG|ZvO&sjlK>yIN zdDuC=pDQ)mYjg_WrYl@^a8hoYmTnWdGSm$R9gmy)`P zmyL;lDV3Ngg|LU<8v{EtFo?p#&eq;V&_jgkKllp1wg2g6r=s`|60nU3)qg6bt*AmF z>ELWe!NbPOYQp)6pMp<-jgv=!pNpG?f{TNbmz_g^os*lDgHw=0K#+rz;=eAcH)+nM z=7K;eng5dY_9Q}O2?jd~va`FpyR*4}Vsmh|VCNJN5MbxvV&~#weWPG?@w5knJXq~r zsQ<-5%FM;Y*~$@YP^%C9D<$Wf3vlB`7blQ8I0Wn`lx7QX*7ueAuk4 zOa=M)KMC;i%1H6?N=b5ZO7lrbiA&4yzjbj-^KTb9|F5W#(+|jUn9|IS(`uDm>{9Atig*E;6x=8+8Ec=@>?Ekp;|8V!e zy53^wpXPrD@7u$FhrgNqTg*GZ1$XIb*xUQX|nqDG3h7gnj>>9{GPR{~rkcQQ`kd zO#dmuqW>fOyUst^1U6s3Xt-vz*4ARbo18P{LhyDwNvA}TK*^zs6=Mb1x;=(sir(`d z4p(>l)lfA+FIk5Br8_DY6dH;s>^Q!ifV9oy63!vGC~Y%!DI9~F|=yE4vz7_ zDqOXw^Atm}+)6xYG7f9IhTAGv-I5|<5bFhQKpZ1>eH2MD?<_W|y2*OKv)g;J*@cha z4o6BVBVnpgJmxa5f(7TJrd!}Vd8h(g$miBgwVrbeNZy)2as4GUf?tHB8-o=D-%G?92iSF^7o*K`gN#C^mI~rb&PTRk` zAIBp!js~%b#2GJmCaW;o*6_i>lK;6Jn$#v&%02}3jm)gH4=DPO(_;qV7~X<1~T`c9C1S&wvq4A0rY#+_Qc z>Eqwk*hnUbhAL;gRmJ`q6dBa{-l68&wMY=nNF^|E)hVgba$*9Ew+eNWH1_eq$RUdr zLjbhvp1U&9;M}cJ7Mjmdb2T7i&{G6&Pn5wl>qf-f3zU{Ag{oVHjURNm{LCKM14eBr zi$hfS_3Nt7RMHj70+WI@e4=p!|Aa{(?&k;Ac%y4K3c?kIL2`LuXr+6bJTKI}OEA(_ z?`9mIRL?Rr8{hf)yQrzTyl<93F6zTD2!8MUAOqEiqYhE!*EY@XS%q(73+~F7{J?3+ zpY40GQE)X!=3~wOf=7cKB1x7r4o67>8?u5bZZGYWhbPqN8rVjQf>MMQI1@{~Guz}i z!@%OXwXSzy2X|lZ_++)MNr4rtKv&YSa@7bQ5#g(1v`E@3Egp>THZzM)ee7cLw_*{R z(ta;fCPlRYzj5axjT56w`jM0reRz_Kn@7&EbHabLjf$_%`Kb_U7_dUG(DwS055T44 z-gj-m_F>|GQJ^W4wGjevNrz@8C4u)kyQnJJ|Mbo&CQmjh9f&=)@fqR+fjIP~N%Arn zYrhB1!$SXDdv#lE1qqX#HfXH-vg^H!MrL2TUQgNd8)$&Lw`~L+mVeQowk)5H#6-?W zl*D}Yv3)ngWTqo852-f^i;cYui?fW5iP%jD-ddp>Y|r0($~bK@@h~wFF-1XGfgO_2 zmEiX}iyEbs9X2)TS3F&>RU@!71kG7Qt9dAlwu?4v^ML_NZ(@v+FmrwW&{2|U5}5zDimbjDRoRb9>s!AgP zhekveA$!U!Dea0C;E3h2*T#KUw3mY#;GoiZ{=Tc4v#+=@rG1%)izz){oYg*EI39`X zo&Ealf@(dvUo|mKStc3ff^HxlW2R-PdtP2=Y^Wfgy!^<1YSnKZJ|d924nv-D_l*HG zDd6*0%p8JPmp~c~9-jBNCqfLR+=8jbS54-&vR(~D$=|(H^eI$n5p049yJgDjg#J>} zULciPmCD{`6NCSLg^>V~RF?j3Fo62_=<<&tLs;~mlrY^g?9>VnsAXv=c`Ah@7$l+! zuu;k)_nm&I%YN2AoAyWWAOX?P%jPO~AT6-)4&ceR{@i{R2gheAqxU-mIP&YuH(#x( zFsM)6sI!V_`OUdNK65m=%-$Y75Svf|By>3J6_mX+qY(92g+)KjFu^Gqp;P349`k}? zp$Ja8=e{#y#gr_I=>2{owBgutD`h#`eV^o9ICV`WN|{yn`^YBLp%6`$A%W4Cwby1f3i)FgiSEc zwIL;Ma_L)*`XYgJ@U%+dNi?HTZK}ckTz9yV!6<_u-e(sVxb~-qqhNqm+$+zP+jhZt zN~7&!`)H}E%|2?JuL-kjcV3N0EN8Jc+VON_yYe`hbF%#G?f6#}^C((}zpb8DIL&Cz z&6%Eao7^gV45L*j%X}+fu#MFo|uc_18#i zntxDg?OP2zrrFOFB(eY;)TSe_^$-W85alnP0Cm@$!X4$aJ`5nY?zl_$K{_2gwyK+SW5f~BAaE6KN(wC{}kay5& zWW6uLR(zk-nemCX`l8vinD7|)Fpr_xrds@Tf`XeByEHbu70fFJHH3nSX!%?43QZNL z+U1(bzY^>|!lz+jA!A{L`G?YA;d+?2*r(t=zqbZ)G3IK>nz)N&Hyp3%K1H<^Es{h= zb5k(#l5svT1pA6N*(u#+lHHM#-I3VCXBt|n6Zg>cbnjel#U4z1D|_#EN@*tYYXNYO z;M0X+#c&`%EE{fd$>xIP!*4k;s3NU)TU4}l``3uhekddiOg#%WbZ8j@M*vzq!bj@e zS*B!&tq2W7zj_|3#C(N7sal|kNd=hW5#=?PGWJ=KYu2B;t&oOvQTJxpHy_1Onz@{*1ykl$nBG3!~pyEu_R(u_hifcE9N#A0~bjD{<6e7&%1 z#2k7xcQrKD*SGhURV<$i)^Q!(yX3SB@h(}@Nls2SbAgm3$kS}zrjL$JIcWMXd-i9G zgh$6pn3pJN*MrNvCVmIBM`#)1{}yQC%%(^f*mf@Z(8;Pey1xq5+CMDsBlKP&(kMTNCG=KT8Lrc(=YmSxGXn+LeQ!S&)ef=v+oLw6_0hXLM-LCH^`U#SI7n!M4LpCXXEy4<<6tGR^+UA(I&=d|7W~ z4zY?8URlJ$4@xRjZ@t;gA3@gAZ%!Nl15YK$%KJ`lwGe|ke%pa*qdHY-6hfULOGEF|} zU#+z`GR%0bZLcU@6K&*R-_jGwvM$wzb)`WHZ|2I>!l3#fxfoqJe8oHGK}k74O!rt_ zBNl~Sxu*3p&G!kzSGvCnTFYNMa|2XUpY;iag)d>iOZ+o)Bt4|Js{(&2=Lud`3NxAN zCBz5I=8w>^0ZMLiH${YowhoqDT4P>bO7hOS#9wSxH4Q=ILvmuZfyRvvcW0M7#q63D z18&VL5mGpo+{C8thnDSTLrEvAbuKMAOkvK=s~NY)>m>urHET`ouK{Q|^W*k*#Ve|S zX7T*z){Sm=BSA0lH=KhPdQb0eh#7e*VQVvo_r)*V*cI=`6QU9GhSRk+r?DvZo5p3^ zT4j-!o>=Ul_t!Ql?u9+88{oP@!Nr9`Q=m@V7nwMjM*G!y#Z)d&Qxg~JIziVy*&LrA zE!UwR`EK{8%+3bct}S(XZ$eI66Wc8O#fz8s5H8;f5|?y7zgJ)LJUWrhM+=>1f}_%$ z47FR<6@9u(#E>0|HAj>qs}lxPs)Gi82PCV}DG5R#@s67oNEQUrH>AQf2knoqf`yVL zN-zH7ul;OP201@vAgn?SDZU+kCKArNfQdK?qPU@B!yGA)VW?trA~k3n!s| z(`8OIiyU(Nym6nAsw$;eHqH88^}5SC+%4u$Rf+N`>kTJp?_OF0ITIVN-}$T-b$?u2 z#ZIq9&Z=~ZZ`bennc2#Zd`_DK=`Feg8ca?eh41tf>|&a{Z%*fZ&v&%i_D21QPCG8$ zL}`cUliEL~XT1H_to&>?Bwla6+_P8}zHhnyb?pinu5L3=l_`0j0FNp9VA-L#Ub#ps z(hU$ptez*7IcgIt0e@zUK&D>TulYJ%giD78ifE;i2-|ykx$Pup(VYp#g*HSPdRVpp zjp>b&MYL)v+RhB5aZOuE{$j?k;LorVq?}SB{4y~@s%7LSos+oJ9Wy$5WZ!xjoA4e3 zU!n}y=!ZBOA;pBAtD81_f=D|D)|k2=AuSz6Q}e6736C4GQxJ zyCx4t`bEHm&WKJ7^(OPlUFTb`6g?yF{Z8;@?QQy(A!8B<4DNwXPlKsvLJ%x0p`64E z!jd*J2qtS`px*5`uf|}1*&<)!1v1d|r;$7!#BS|<2W>U5Lg`n*moOqR&A=jYb6tW^ z3}T@B+_q+CC)^GK!ee3q4sKq2$oYbi?9VCgq4ZdY`~}XihFv4$KI?2_TJ)1QlCgO@jCTz(p`$hH`WOUZgVo@`?*wEcvUcYWuEcPwt@ z+1ox+Ka+5x>ePQBuwju2U;amZR-Iau76Ikr#)*^D@TPToN7ujq>VO(D&GpB5VUm#= z*P-p^A^j~u&!Cg-Hl3MfZ=bmAW*3b*ToCw3&iYECWyo3O1h%)|tY!E3vu5R*tl=tO zq)(8^Wo7INl*pGYSG76NZ8QDQ8J3+r>+R6#=WDIYgh3`8OrPype6+qha-Zjz3uNd5 zrs=mL{*s88UH6Nkg9H=*LJ4}f{v~obu(e~-AMG?T9JI{CeWKR8^s%bV?J+W$`EfWr zH717lcug;tjK?SSZgc3D%|}931DtsQ`j6<>=uOtgX0FO&H~Zp{rZ&qL=Fz$Db$Wl> zAK&qgodW<(r!QeGr%U|cHl=dXNxE8$=p13%`K`;utW=ftmFVS?6&1ImYqg;(!L09S z>;AcCSY-aYX_Vd=B*Du^u+nNf=bqmu<}S9y5@;a!HgmR6o4za9;#6&ZeWhBJtCPK) z>L{*#*=$w_|GIX)B@EuXbZge8%iVU4UbAk(6^!!kcSQ=`fw2ObHzK0jwPNTGa@*A| z#>1MQWMZQ~zqX^?9pKDWx`O(@3r06lpon-MOd8kX`K4u_1ikQ_ zkWIVhEnUxM!90BNo{8o9#APr!DFZ2hHJ{7|1Ee^`o2qw7t=)N171B=@HI!+wjTvWmw>C9!#|y`67C!;{0cV5K6SgzF^cS3UygX@^H#tuaxXwHWi^w$I8X~cVQ?PBuPK9N;Ht0 zHgV>9_GJnxBf`718F2{@x(bEjh)kKqgUJcd6zNzSwxQ>DaJ?Rnm0ZFCmmEKSY%>Uc z=)=WK`g$&bn(cFjb#t1BWE@QiiRV-)O=i~rAVrfPFBv}74?4P8)txGNT+;ZQf?^D2 zVq!a2scTEa(KrV+F7s(=YV7#d``ourZuRZx+PX>M0EXhodR=j=coPF|a!;9-epx8f zEDv-oO$DaxjSkZXq6rPo7q$!E_LAyytsoh|N5&=Pw=^_daBm$;9f`tR+r6$oH!uF2 z`}590T-vwtp5*~;LNrgV6kpotg^^??>ka(1N> zqsyzR`bt5P-BR&AiHpSwOh=!dtDtpczGnerV2u7q5|B8R9sW7k9uR@dqSe)+n8Ca1 zb=vlMeY1lOAR)xcRGccHe``@b!uL+Q zPXDVERF95^3eb?lMntemCvlh6)4k=e=Y|rjO~} zZ}DGAa|1Y=au^;@XP&-}{02BdO1INMQexr^4)7V))yd)~*p5#tWO4J>KgG*bRk@LC z_nC;#cPle+pG`AGA@TxQf;IBlvUJxDo#$1MJ^)+dqzW>Sca5tVhX9HmB8wq*eEV?7 zWNkBNbM3`1cd_QWd}4e0OZ;S%rRa(jg>ev=Ce`4&7i-tJD&Q(dBj8>=9`<)pU@UIU zV8oBdZkxUNW*Sm>b_T+aLAU%Hzsg98L&UtN)pg<$u|_iz&Bh``iTzqjLtl?xg(@j~ zSbO-IUSpw$4w=^2pEg2=O2-b!iQP=O99ZF?eau9Ic8gERW66F$#yCIUCIYJC1+JbU z98~xb)ByC*<{#-u(7LKK;!mZCH*Q2p{Te|ul?$dv*HsWGF%7(L4@9QfW z#PtE`bD4VN;Qi_+b2N+4A+Yh?CPrtBH1!@x`K@8Ps%VTwJM7qL~VlV$ruCje8V}(iWht4no*BJB( zse?RPPqVyxjVa-uS>Aq$y;6}h=VaC7;x#cl`BFbG5eyIFRL2G4QeO*A}SimDdc+K41&rSiV;5fP=Pi2_*-KM63G@_>4JnX%8H z8znTw+i&@qC-qpXfHsX&_$tES+`~-%6-p!PS76mQ%M0#qL^PI;!^q*dVj9|OG0}T; za+i>boQRjN@HdaoDmFaE>H>3DJ-SY@mHqL!(q8ghbjqm{Qv~8}MuHoM*s?>+a`MD^ z#U=o@)c02>-l@?;YU>Lh^t9D2^*cOWGK+d;`oQTnln#}^p~+gw_rv}mf#o`nLGLV` z!DVIaAR-gDFqS+!e4`=6NNh^1tAgI8u7?ddwqZR|tNs36z|HGaE}miJrsR7xsZ+Mq z94kv}VH}*jLJgy=_N;>$0k|WoTuX#c*g6s}HU*bw2|*T~hHJ$!O9ycy+#OF(QODiC zERtyy6-UMGf(dv_2>awJJp^CA<+QeS%&PQwODe6746yM_(bVazBpESq9&Xa`SBR&6 z)bFXpLd2W~T}dAch!LmjS>_JFAe1p|blKrZl6L;Uhb8tv_)-CP(#$X8h;pHQcgXxv z(A*Ry?`sWD=!6Rwv5HrZ3Pum{?Ms28Af@QoFx|4Up)$og77wenV*5n_kbGu0S2F{t zUsFjm(K}D3o6Owl1Nda%V3*gQEX?l*C>UfCVM?TwDqSsCZSI#)bBmwJPg{aWY?nM~ zqepC693YzVrEr9<#Ar+AW?Gil#BB6(!X;3=p4)NhgWt)+-w3O+QU>OJ>Ajdh`63(< z7t=y<87_4&)xp&R;g=Nqug&7$Ov{o&aPumixK;=>Xn;6Si3Q7OW7O)!;};Bq0HS$o zMx10V^#KM4OLy(0VE?3!>!jrsm&cQxQFE6o`eCs+IeNyY7y29`*_~LXrl=afi-4k0U zB_?|u@wu5BB|*_b%Z$m~)Z5_h6BXpZCPGe6LILJgh&AxWVf=StQL(koa8vieXWS46 z`4I#Y(`_uT%yZq<;?WEtDQTI!1e(Dln`%4Cz)Ti2CaQW}vs4Ls^akyG;mT>`&@o^+ z?l9qO*Eqg|{P6JQ-@Bt|{UA;h@f;(TsS&^9TGuPc+h!sf8IV!#C4nj#pz2ni#Pb1C z$q<@#VAYdqOqFP|rD~uhd7%|>U(NcV>1wpmuL^F(*?3X{t6UFOknhvr>igm~y-!=& zWyq#%$r*fhv6i-$QqI=I#p6Cr{omfULk3`X8~A`(E%|wxjL)!`=+W>Plkt*xn#mQm zG%u{dan+_-#%z{AS1;Q15qaCS;~~fH6wPjKAAhzm2I%39q#eU-l~`+FgeU(BvWAFz1z7f*5FUrAw`HmTC{;=d|~Kz684{0O&?s1`3ab0{WY2RGvy2D+%if}DJ@b* z>_=H@*R42I#;%>ZU+*go!@6muQ{8S2cDI9r^ZGGpHa`lP z7j}y$1poT6?zS!xO%YN;;W@d4e*f5bveH&mIn@6nO@uz0Tx)7(B{;ZA%66eu`etm3 z_WRZk|NdKpDlJ$cg~f#p9&J@g9D})ibYTmfxrGr)a!Fv_wYj8i3%UHbB(*zK zld&yX747XVGc97r`yH#&svG0Md>lMd=gnd$(6QU@ykdSCr~)o!Z{@VT4Mxes{L zEBaE5ym`D2EOIa^M0|oZ0@58ag>|kt53CMi$}cW9r+~n-*&+NErxc@xT3Mx(vHq%n zXQk)HqYGX0wzH?){ialt$%|mF0Zsso)yEwb{-<(x2uO-E%{aah)TUdhevEt{L`k%R zRe?0Quwcnh{v&$*m-lTv6GV)2dk27;+q92Gr_{9c@JHl!G4A$)8)NaO`Phok#>0_! zvHN~Fx?Urw3!SfcBB!&mbyz8&CN9w!`9tX~ClHbF>nRz4bg<0o z6FFI}>BdJ`h!&@Ktr7Q#*HwIS!memL{AfdI`t8%tqUmC>)U?SmxY+N>8{eU)L``?s z^Qo+~FV+y#qLRj%K52isKW zwaiLirP)W4)t*6mUj9P46f$wf5C2s7s<(b^)_+$%8fJYl>|7K$g$J7@1g_5>ly25% z1B54&76CaY3tKk{t?(e?-Rs_4qbFJlO)VX<;5zk6V?+)b=1;f@s;XxYg6fJNrj^uh zn=;|yF_QA~C{UyH2Sz?DFbxZY#dmdb)bOj`8vF z`UuN{D?;yg%d_a+lb`1EJ;=0;tQsrV1`7W|iUu|OMtNEd{(&jF1 zz+<*3<%rL2!C0@sr&;u;34x{$6ouUPCe5rXS&Qz)x>>ppOMM_Ie1gI0&sXZh#Xy9t z6qzdA_iE}e8)~=wH8evjBLoOroV#^s>RL0DpLdC|)C)_|YEs+6RS-F?fLbbl>4*nv zZM6yWq;@7bn3(kARYeP@_v61qykz>w@bU5OYjstX$ig&2(4xyau$~{U{&pM=v-9=F ztMbh9bt(6ObDKV7G}v{=wjo&31;%FviC6a3Njsbf`}c&C`xP+*tI6;GcC0TP)h$jK z*9NFwx5bHSirmdqWXfh|Wz{=hGp#svU(fl!ZskCAgx=Fu=5t5IMn7x_Zkd?$)`|iR z=?IW;jJGm7u5`DZn_9dtl(BI;iGBPEj7%}RxPASo~ zey`Vjo_xw_5b?`2=N}DkALb{G*9%m35)^L{$Aqi8DGU3>U#K=0m$i*eSQt4eG*En< zL4bCn@s41d65mzuN{{8psL?OMYULCX_2lFRP5#0C;U7O3jI3aixSVCSQwI}TpO!E#aK8oT2||HQl9j)2AvFKh;TprzEL5|WHgm~sF%*ng5G_#!pW|=Y zH|LY`fwB;Y08}$;GHyloeIF0Y+ZZ0vpuLE~l#osk5mKG9ajo@{0Rq!j=W=%L9W|`H zJE#OaX3(oQEpypvT<;sn<}9O-ZF*@=EZF3@t8h4LNN~X?Gg_&aK&MTuG{vMufh_PO9hp5}r_TEwb^fEV}r5 zus3W{vgYO8d*4hAC2gcSe&l6$I6XooEMDcXGJacqdg*H2V{ea{+W=SkU&ru$0^~E0 zuvzSDPd60W^)8um%jd^TjwacpLR}B2^5lLmNupUi+?z+4J*O~t#2VCEyYkvxf^r50 z9n^4ePjIU~Xk`k4wM*@OJr&wfR*cg6Pt!e?82P1X>g(b}HLL&~CYr<0iS#{JXXkl8 zTs`2Tz!8-5oMt~-fB|xH7votXP))kz?5hUYl;tcj*S_d6rkJ@`t^}DPb&SVZ8j#1m zQMG0($7`~BB`*rx1MQ#0-p0wFns1gwu3krIR)Eday#1M7z)|0V6)nR$5(Cmsmo&W- z@1bR&pf#FU_=8t&W~`A6akY~|g7t?tUvt4TONQmv`N45|ARwJs-Cpb>WVhM4fIg)_ z-2AX`lFyaKa;ac*Y6`=1)o%WvrF41(hPVi=-e;fKAU^{`CbMnu0TQ0b*g@BzgC!R~ z?y)cc?_*EaQX#N`GUS|P#1-0qZUHdfIWzEF4gYx!Z@GU_eLm z@ZDrP7h_Y+%A^TQZI#o}!Ltv;G^8)T|Du;cv&h(%4(-`p2u zovw?9YU?i}-6%+uwAeHY4IT{PjOuvO%n|A_DMl>fzNFIlow9?~$e)UpS$EWW&t>V5 zk9>)5_D90L#A@PEexinTU|)_+F)~2Ost{{21>+JX1cJ&wFm`Qrh*1KT1GSIRBPA5B zf^giAh!NWeP&~1$&?L#ten%-MQ~LYj8sln|R#x<9>ckN;C0~Gie(=BO#fzTz;Qk!n z{+(?AMYuU$uU>8QXP*Klp7m-SVE{#g$ryn6^7+6cYGe<_- z#waJCOmg%0%DD3hc)YSn-BqgUTnG<`c-5=*Xe6w5Qi7f@PzTar(4HqY_Fi7k8baeRLUmZ7`nQwbUSiVjO^0NLS)fx?8pjST zZ4qz9%Tgq}D0lY(O!T?eLo3cU#z~Z!1N$NscCfflB>&5v9Fk1XaFdF= z=5A{5-~r2#UKBL;nw3!#;aRzD-q&1lZ5|_eer-@2GcFx=QGAj;VL)uq?A|STtq@6U z`jT$Ms6GBEx)zhh<}SF5ac~34E;u0{(MBzHQ|{C^<$ lM}_})G5&Z@9WZb(B1%Vpt0E@M|2drvkXDkalK2|X00amlZth?Tq>#4 zJ@}Oy21ZEK4Gb`|1wtuIftJ=FVXBknb}9;Mb73lNZe=!Quq4pRTFwIk)bLQzH1n`E z<2R=g6`>Gv6L3Jx}Q9#%GfR(39CHg*9vegQUiihn&+uhJmq76R&0 zGXIkG+7qU-fWOd*TAt|4|n{ng59QFJANitP9(Jx{SD%NxI`@OsIXestMXJ1%xTf2}p*NzN=FIGMI*0d^BsR95Rdx`z1O}hx)7}E1w9Ifn}e)3*A;rx_2n3BTjd2)4JJKKCMVoBe;%tB3ZTORO1QcO%o?(g!yQh~`UUhc|s z>BIA_6{NPsGj|MG@Sm~cG4Vvq`#p)%yJIurH4N0%jf~LG_n=KK_6u+baC{jm5>j1V zp}Q{xA#YwfTAo7DGx%_mGgAO;MEqlv?ww?llfxo2D9VyUm6+5x~4=wNUM1+u&h*6#JeQqCOsr}zPb13Y&G78~10iFSlaYo=3_CH4oL zg{G1yvA21oocj*2E~(bjez?+Vd$Q62BU+#_ndRRmdt;X|HahQ)KmO4iDqq>RXTH=j zx+I_SumrEzc_D$<%;TCSV&HYH;BjTBP1&Rw zwo5efwvKoh_nJN#8TobQ##eJB51ZEM7-e#u^}c&gp%6w*!{kNpXSun|(I6)GK6%7( zFYa0o;*>Z=xmoJbI=R@tlTql<$_04a&1Qb|t)k-DtnyN-V0sW2BDe!?+AOhI}=7O^mjw^?3W?ed1$%RFivsOt~>&~)m&96%?1i?AD zG#LmtIUkS^4px;Y!d1Qzf2zRTk~D$80R7p@wKV_DU*-vg`uU21#Y{k;vf-lzPcQ~{ zXH;ygLSB{bW*?+6{TC4F849@e^`t|grus;?(;S(m($|}qkfg`-hcb{{x6y6B3pnhv za#j7kygZ}2*#JZnjyigKf7^4p|4wbHMVE5}{hj%XPpj&Az`JCYR)Ou(QqgB5WOba6 zIb8VI@2`Mq0f86%Dxm5{HzO4_H+nb19MwUBQKWzNo4*b$eB!z zqJd!F15l>=p~Gi0I|nZ>fMq7Rjm_sVV7|ITqAJ?|XFQ@M0A#eXZdOSY*=Ohcl^D0u zzNV45$S^%-aDViXUpkxdcZ<4t`uWnx>C<4Bhf+k#ok-j@Vs7ZSdKqQv zKqh)7A()6t)_NX$Z1KnSO<@FkU%Ox zg029m&>{}mR9QzB4VE~YtF6q~#00<*uO5@qju-*PL)G){%Xe5Ts@*s1>}5?~-2Qy2 zjE*Mit#SoS&Lmv^`pT5)%Ih$do7J!zOeZrZiqiQly`?3-;1kMX9ZA`n*3`RO4}E)n z8GSp>cfI0H$=N}fc=h}C7z!+1xFwYHNbx^i*@ zz7bJhw9D*%x4?gz^{TG3O=cNQk);HVb8 z=JQH;Hmtr$*?#05s6H_JsxOtYF5qWw!zKe3D&hq7% zRL~dL3zTfLcnZtMAQ#M6EAIbFbho7O2_7EdSxYC3t^U5ayqt!12Fh0Brpvs6aKFeQ z9NTa#%R?8}=DMOR3Ms(CI@GJ(Jz~(VFGUz@_|(N6 z9^NKVp?LDACS$NcS(lwEV_s;@r?;>{4RvRsr6|cvrS`ecqCuTjJ(2}8rpTS7jK!BI zB_(5*I{z3NzDDOS=viP}AjZ3ER)hV^Y1^EG}9HZJ*OOur05 zm;@hpVN7#!a@l0`S`f4Iy*(OYX|izDAzYY#FF?{#&brtbZ_}@0=J)*%_^=YUcy8o7 z_jLSGh^h=5awZrZF}-sO|AXJx#DmE^Adq*86P;9e$dP2kegAO&NYJ(Iu?xlb-5WUa zKzaB&etyvITIbV44<9MI^|@IR9v*%{?h4=c7(W0agT~a$v z4vTtL!%B{`sXQ3p7mmQ~J31=FD{Ea%m&yvJc;NmAB)(AF9;ELFgV!}&q4mjg(V|aF zE!NXt#v_x<@^0h`8x8#Z{R8i>XKZR`f`$hGlA=0QHB#k>;`K+j*1y*}m5I2E7~U#e zc`%r`?Ln=&y*b|$k6P5~oxwT6WZsey4XCPR<9O@jz4TpnXls%+&T(zJCohuDRCZ_i zJQcN?bET9SG>3>mS=g}(7&!?nMl$N3Be>uAxZdzufpy&*D|`o!t`UysWaVE>mvXsq z>N-*m5~>s@l9M0Y+}+*UBLyl{6%bvYt`{?(9wrFA8@DUhu}+l9ZJf=4_yu7zghebM zt-g6+HsHLzfJdjPhQ56wPj#qWr=F!qqkq8VNJt5j_MnT}=0qq=?O>{KI-T%D$=wp+ zEpM2U2zw(uD_8*i@g zZ}0(P+{RsYdo!(`X|tuGMtAeEzjc(_N78s-YTYkHSKV?_2hZ~8Ha1i;_|r1t1NV>@ zvuFucWJA$VKoeO!LtT?(QIX2yBO8Y!bN* z2eDO=&Hl)Xpia9C63VK|lX|<@sZMlz;*k`Auta^1>TSMv?*`pIvx*mdOiLI)&vBJ8v&X zMjCE!N!t(8&&CVcg*`tD^OL+>HeMVKnD^=pmOiy?825O4nIDf3s*wtQ&buutZ&PW2 zGKf%8HaQjeKi!W$nXK(`gLsCb#*?G#f81bYxl zXBY1ZLMDFw`n4KOc7%t^?Rn27kE+E?IdhpKYPb5_+jQl2r>Cwy4qDc$x6^gdvbByq zDllv|k>J+!PtCZFCTNGTcbU{`o_1M1;^I8pt* zGsCH5GUV$zKlJjBJ9Wl0+t+r_Hoqy#b(x@o%QQmKAuWM`^N4VZF=O!AP@-r~#jx|n zufiWcsA>|i4b|jSF{`kIT)#N-f26aVeT1KHY)m8=C)Q(*TV^atQ=hxqZQ)m z{J_o``rC2cYIdftd@pD0K~WVUXzYw(3VZ~f8$ID##SY8brAchT=%C+K+Xq*xj6H@U zYsRqSMej>?n4~0q7we4+Ra2Rl8rbbkJ!&iFQLg&D{$v_q=4&%phe+emmr!qpsml*; zYdJug-2U06HCbmnAhQ1}i90p*dwIFAVI?$?!_YjUeTUXE;*N7kI$+B;UImi0DL#*jy%jn%*mDt#; z#9#GD7>H_^?vt`$kjZM>fYf?5K8vmgH8Wa^tj4O>6)qpm2%;4?-xceeufD2g&=#a~ zgLbt{ml<9+ZM1GTej98!K-^i~FC!BElQ|b{W4opOx$Y)9p02a^>xj}I)nf+y*pOHY z@Zs8F*0*@p3LN)&G;KVMA66r8MTfqZB1~C{IuF}BCzEKIEL8C$gp@^anEpgu+#Ei+ zTnDrQt+6e(!7KBt{xrT8SFW#LqbV(AG7w}W29E&8_VjS@Tw9n9*m{jw9V{$)ZE>7_Jg!)Cw6>QXx0ajvT89h(P~O6go2#UtA~VZp zK0!eu%LqBVbyVUUs7?MLBnQ_UqTHgBOIUxaR=h-AcQn1Fs_=vhxzt+Qa#0nFB53z^vTR7;s zX@?gTN6M(+`pB3-v@JDUmbX8OtQE31rNCnAm%dH*Bj^XbM+P5&J=NgZN6Dy-lYO#_-~7&f zEQ_K}=R|0O?2$ngXE zzcqx}v8SOZD41Y={XkDY@;n$H+tEo!Yt%xsyvQd|#5v;USf0AkQ6S^IdrDeG)krc% zP)OD_bo3*Rdb=<3Kwcin_q2`e^%H3$=JA8|W4E=FlT+taQBKGQN~3lm>mM>u$}4ns zPdvQU*aw>)u9}7#9B;V1wMkFt5o7J+xJWpzgKB)7y`>HhE%T@e*G9$;t5J`ocorQQ zaZYJop%yGU?HEjt6+M(-w_v$Bx%3fr(@!JP*Jf#JC-Z_0b1o$~SXz3QvMFbpo=Ok- zt~Qag*=$@0Q@l==iECNWFr*IJ&+)D3K`LrLv~PosU{g73U2{Mie@RcF*_v zoNitx2)psg$Md`}7B$tI)#Tst1tA96AD+EOZ>dH` z>7#tb?4_*QPoN8h_T+Bzk4HYgW8>IX+d>oCY7N%S}c#E`HeD$`Yt#);WtF@(Je!W zU1Bsi2+oRi1aZXZ655-}n#yBOv@Y)aN1Uswt%YP#wj_0^mMHI=!Td?a&YIRKM$uWJ zV;d3?s-!NH#01IN&+&?NkqtvGLy8b6cA?AWcCdI3e^iwo+>$Th#Gc)3!`WtzY$sYn zZb$BCUaJX;j@8xGxw(Te=eOn-dbLF8FlCUf3c!=c4fg&FH@=)3S2#T) zc6?k`GJD^2e#<-MNH&_I>HxGUP7XYl2e`XkL)YVu$J;f@1sU|0dcd?rDsa;A5+^_m z(g9)j4X(^|H)Z<8z9O?a5A*CI6uJLP{%7mMEfQx+K6{1`;YDhdR^DgPr=H)iq3G0p z3B;)6Oh#2UcXVj`Re?PkcndvS`0vFi<~3Oq+76*hs;AaJL?w_vJq@T-$iOO z%2L_lUIqc7aLU(#KB1O*Y!$u=V`D(hPl-yio$}Yi?1|oeduCj~VG)E`=XWm+!m*6& za0&!)`snqC@-V)GV#&Fe@TfgD^LejGE3FO`xG75?g!7HhrzHWSqyFk>8KqV1xx)D& zYU%N@`5J4b_11+Pi%SNb93lBj`66=?f~`+(*3aF8leWOrJ13704T{$6pVkh_gcIfD znW+1V`0;PD5u}!0qnG7}Yj+V?VNwxtR#tu?0_hZ`^qKGprRuL6ZIvMa;D_pi;c`V0 z9Ubsi7#4eVZqMD3$5V?td?*HSF%}jbXIiizABDkBX*Z*aQ?Pi6Ycv^i53vLRYKZi( zNTb(L>LyPhd+j+ga@%6f-jVB0OZ}8C2p7A7ukNPhJW>$ekB9l7B)o#ruT{fuAL z&$#j06Wbe!AE!2Zrq!X+S51O+a7Z_xT3M3#fejPB>OG-t?(Q2s=&`Y<({ftcC3=zTrQ=~=XW`)BtTd67 z`ZxlG%R2~&@N!LNrW9)(WAP7yD=YtaBP#2mJspKq1tLaTpct#}a;3WGqI%ZTs*J;O~s3pWQD`=b|A`TZ|qr7i;%l zADMhfsJXl_ttC-w>oS2^$LwS;MMU5I4!)~;Jtd#WM#R67}#HQEDhRvNkgMBSOJM$t)f{bIailukQWKsTTmnshT zFrzrFrbIOcaV`xEh>@Fl+}KPvadSz%h9W7qPP)7frbe_zjR7JGw75OT(JN`Z_G1P$ zB#8LFRW0_mS-4|hVOnkm+F)0KNt)^z#ynN8L={T@vZkN6>n)A(M(((FGJ%~7<=WoI zgW-7vOsIGo&*$ODIl+o5tVcx45_uC}fT^B_7H_3VuzT(wD$@-oK!m#JVb~2~efyv( zB*suY?Gf5e>3M$F&mYX0nAUBUe#RCJC3U9CS85(PMFcMhJQ-jFt)KUjGv7g_<*gY) zuGZGO5z)z$n3&mNkP*~m6sr9uyF>fG3;GJNpkF*sOy*S5eL|k$3+*=HLO_I%bcgo5 zUS6soTD;QMYvMwb%vUQ!z#u*Ex;=AcM!z{+ zc?#q|DZ8^^Hh8`EtGVgud}=4hcA(o!Y6(#(_I_EO*SfK$R$$U)^F0esL^_jWD=Wx1 zlI)`bEnptf$8+2L2CpyiPn9`SB4WE+m#Qh-E;O8ZR$Gqxyub!=^n_~^e2OL&zJDH^ zdc3<3j+FqzoZ({!(SpO(DlT&Z^(FEWQjkel^zCUSYgYH-fi6ctV${pL17_lEYNpuRTb}xekEp0}c>_`9!Z;mPyu5x5 zWWyqUrZbKIfE@hjC1L6|oj)%8R71CXvx`n?6&`bR{QHe@sk-$R^8JHWv}_IfD0BFD z`|6+16OzSw!$Su0gV$50=A)YN`RAWYO_*0~TD9`uV70dH8a6Xx)>_NxMCvr}JHf@; zIIr1s&{KGe8+D0HTq%QJlT2>AS03~%1O)!?%C^f^P{@!{e8d5iaUNV4x|+kMV0l;~ z<^1Bg^6q_;Pct`xsR4}m5mC;D$$>;aQ$S6aHUN7So}9VPEBrEj3yHL_?2af4CRt@hH*^;h4A}86Cxp# zwex7a!QNq6b2wkviOg}v+b^B4_h};*8G4luCi}g^x;D-~zn8svyX)Os7`UnYA$k6= zPm+=3vpVLNEImn5e!bYFV^v0q`QK{YAw@mT08_HAu6@8;x9dT=z`y|h(AJW!=Xq^9 zW(&{FtZOFfc2+E)R+&-x=fdl2i@5m4@2*6N3Vr50S7A|P6ZPdgg3s@sjhLjo?6Am0 z)>8_y-__V$U*5GO8uxM}7;hrjpZ*9{WSCLlH!^xn#cdwc=E+dQkrb%e8hh_ao8Qpj zy~mp+N_g`#(}$fMA+WPw|EYUCS=;+P@!E2gwVZ9i@L03^_#K^Qm$wyC;EZuX{>s`f zo3<{ibguQgXDp)xaJK@ zBdy8RG%gH@33|$T`3vp3`l)gI6V`j`-h)5asf~R6AK#9zVmY@Z7egHoc284IM#qQ8 zZ>HQ6wRM`INNM*9^|SVo9!_<=`&-MrIsBTWJqsNPUB2trVc^X<#Q;DuIJ6B001(CH%8Obx4)MQxZPG5j1_4T zr7h=uW=!Lbt&cKE<<^2js3PJX9RQ-MSa39=ppfuRHRRHG8fC|4GbF8lRKg{S^w_vP zD{p_SQZJ;zc}?+QQYpC7EHqk<_mwWb#+I6lc?M-LJLxQdf-D7j#b%3`KVMvN@bI3t zkyC+7c*Vg#g1m>zl`?-d%VIF#|BaA^5pl!*h7=_{{PoJF`Q%1&FD1pLIWn6t=49a9Xx_CXRfC@@MF(&ku*3E-Rt$F__v(%3!l*uOAPiXmzXPZqWFuXo8YQ#Q2< z4?gP`UahULux|;@WE$1?Vo#&9GIezbb3T?^ls|r~j4Xf4aSW%WHh$yvfri>GMob~Z o&#xlp@A5xet!3VdzCnP&ExZGbC_?1_`hQ1WT1Bc%!X)7T070y!VE_OC literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/pearson/index.html b/content/ko/case-studies/pearson/index.html new file mode 100644 index 0000000000..ddb567afb3 --- /dev/null +++ b/content/ko/case-studies/pearson/index.html @@ -0,0 +1,87 @@ +--- +title: Pearson Case Study +linkTitle: Pearson +case_study_styles: true +cid: caseStudies +css: /css/style_case_studies.css +featured: false +quote: > + We’re already seeing tremendous benefits with Kubernetes—improved engineering productivity, faster delivery of applications and a simplified infrastructure. But this is just the beginning. Kubernetes will help transform the way that educational content is delivered online. +--- +
+

CASE STUDY:
Reinventing the World’s Largest Education Company With Kubernetes +

+
+
+ Company  Pearson     Location  Global +      Industry  Education +
+
+
+
+
+

Challenge

+ A global education company serving 75 million learners, Pearson set a goal to more than double that number, to 200 million, by 2025. A key part of this growth is in digital learning experiences, and Pearson was having difficulty in scaling and adapting to its growing online audience. They needed an infrastructure platform that would be able to scale quickly and deliver products to market faster. +
+

Solution

+ "To transform our infrastructure, we had to think beyond simply enabling automated provisioning," says Chris Jackson, Director for Cloud Platforms & SRE at Pearson. "We realized we had to build a platform that would allow Pearson developers to build, manage and deploy applications in a completely different way." The team chose Docker container technology and Kubernetes orchestration "because of its flexibility, ease of management and the way it would improve our engineers’ productivity." +
+
+
+

Impact

+ With the platform, there has been substantial improvements in productivity and speed of delivery. "In some cases, we’ve gone from nine months to provision physical assets in a data center to just a few minutes to provision and get a new idea in front of a customer," says John Shirley, Lead Site Reliability Engineer for the Cloud Platform Team. Jackson estimates they’ve achieved 15-20% developer productivity savings. Before, outages were an issue during their busiest time of year, the back-to-school period. Now, there’s high confidence in their ability to meet aggressive customer SLAs. +
+
+
+
+
+
+ "We’re already seeing tremendous benefits with Kubernetes—improved engineering productivity, faster delivery of applications and a simplified infrastructure. But this is just the beginning. Kubernetes will help transform the way that educational content is delivered online."

— Chris Jackson, Director for Cloud Platforms & SRE at Pearson +
+
+
+
+ In 2015, Pearson was already serving 75 million learners as the world’s largest education company, offering curriculum and assessment tools for Pre-K through college and beyond. Understanding that innovating the digital education experience was the key to the future of all forms of education, the company set out to increase its reach to 200 million people by 2025.

+ That goal would require a transformation of its existing infrastructure, which was in data centers. In some cases, it took nine months to provision physical assets. In order to adapt to the demands of its growing online audience, Pearson needed an infrastructure platform that would be able to scale quickly and deliver business-critical products to market faster. "We had to think beyond simply enabling automated provisioning," says Chris Jackson, Director for Cloud Platforms & SRE at Pearson. "We realized we had to build a platform that would allow Pearson developers to build, manage and deploy applications in a completely different way."

+ With 400 development groups and diverse brands with varying business and technical needs, Pearson embraced Docker container technology so that each brand could experiment with building new types of content using their preferred technologies, and then deliver it using containers. Jackson chose Kubernetes orchestration "because of its flexibility, ease of management and the way it would improve our engineers’ productivity," he says.

+ The team adopted Kubernetes when it was still version 1.2 and are still going strong now on 1.7; they use Terraform and Ansible to deploy it on to basic AWS primitives. "We were trying to understand how we can create value for Pearson from this technology," says Ben Somogyi, Principal Architect for the Cloud Platforms. "It turned out that Kubernetes’ benefits are huge. We’re trying to help our applications development teams that use our platform go faster, so we filled that gap with a CI/CD pipeline that builds their images for them, standardizes them, patches everything up, allows them to deploy their different environments onto the cluster, and obfuscating the details of how difficult the work underneath the covers is." +
+
+
+
+ "Your internal customers need to feel like they are choosing the very best option for them. We are experiencing this first hand in the growth of adoption. We are seeing triple-digit, year-on-year growth of the service."

— Chris Jackson, Director for Cloud Platforms & SRE at Pearson +
+
+
+
+ That work resulted in two tools for building and deploying applications in the cluster that Pearson has open sourced. "We’re an education company, so we want to share what we can," says Somogyi.

+ Now that development teams no longer have to worry about infrastructure, there have been substantial improvements in productivity and speed of delivery. "In some cases, we’ve gone from nine months to provision physical assets in a data center to just a few minutes to provision and to get a new idea in front of a customer," says John Shirley, Lead Site Reliability Engineer for the Cloud Platform Team.

+ According to Jackson, the Cloud Platforms team can "provision a new proof-of-concept environment for a development team in minutes, and then they can take that to production as quickly as they are able to. This is the value proposition of all major technology services, and we had to compete like one to become our developers’ preferred choice. Just because you work for the same company, you do not have the right to force people into a mediocre service. Your internal customers need to feel like they are choosing the very best option for them. We are experiencing this first hand in the growth of adoption. We are seeing triple-digit, year-on-year growth of the service."

+ Jackson estimates they’ve achieved a 15-20% boost in productivity for developer teams who adopt the platform. They also see a reduction in the number of customer-impacting incidents. Plus, says Jackson, "Teams who were previously limited to 1-2 releases per academic year can now ship code multiple times per day!" +
+
+
+
+ "Teams who were previously limited to 1-2 releases per academic year can now ship code multiple times per day!"

— Chris Jackson, Director for Cloud Platforms & SRE at Pearson +
+
+
+
+ Availability has also been positively impacted. The back-to-school period is the company’s busiest time of year, and "you have to keep applications up," says Somogyi. Before, this was a pain point for the legacy infrastructure. Now, for the applications that have been migrated to the Kubernetes platform, "We have 100% uptime. We’re not worried about 9s. There aren’t any. It’s 100%, which is pretty astonishing for us, compared to some of the existing platforms that have legacy challenges," says Shirley. +

+ "You can’t even begin to put a price on how much that saves the company," Jackson explains. "A reduction in the number of support cases takes load out of our operations. The customer sentiment of having a reliable product drives customer retention and growth. It frees us to think about investing more into our digital transformation and taking a better quality of education to a global scale." +

+ The platform itself is also being broken down, "so we can quickly release smaller pieces of the platform, like upgrading our Kubernetes or all the different modules that make up our platform," says Somogyi. "One of the big focuses in 2018 is this scheme of delivery to update the platform itself." +

+ Guided by Pearson’s overarching goal of getting to 200 million users, the team has run internal tests of the platform’s scalability. "We had a challenge: 28 million requests within a 10 minute period," says Shirley. "And we demonstrated that we can hit that, with an acceptable latency. We saw that we could actually get that pretty readily, and we scaled up in just a few seconds, using open source tools entirely. Shout out to Locustfor that one. So that’s amazing." +
+
+
+ "We have 100% uptime. We’re not worried about 9s. There aren’t any. It’s 100%, which is pretty astonishing for us, compared to some of the existing platforms that have legacy challenges. You can’t even begin to put a price on how much that saves the company."

— Benjamin Somogyi, Principal Systems Architect at Pearson
+
+
+ In just two years, "We’re already seeing tremendous benefits with Kubernetes—improved engineering productivity, faster delivery of applications and a simplified infrastructure," says Jackson. "But this is just the beginning. Kubernetes will help transform the way that educational content is delivered online."

+ So far, about 15 production products are running on the new platform, including Pearson’s new flagship digital education service, the Global Learning Platform. The Cloud Platform team continues to prepare, onboard and support customers that are a good fit for the platform. Some existing products will be refactored into 12-factor apps, while others are being developed so that they can live on the platform from the get-go. "There are challenges with bringing in new customers of course, because we have to help them to see a different way of developing, a different way of building," says Shirley.

+ But, he adds, "It is our corporate motto: Always Learning. We encourage those teams that haven’t started a cloud native journey, to see the future of technology, to learn, to explore. It will pique your interest. Keep learning." +
+
diff --git a/content/ko/case-studies/pearson/pearson_featured.png b/content/ko/case-studies/pearson/pearson_featured.png new file mode 100644 index 0000000000000000000000000000000000000000..6f8ffec49e6efbba25af84b21319bcdac403a2d0 GIT binary patch literal 7784 zcmb_h1yodDx2H=%M5V!DKm>^yYUm!28j$X0Xc(BGI|Zebl#W3WB&DT91f{#XLmKIN z%Sl{o8f+xpB^`7pl)mNoYy1u&_uK6`&fJ`vm3;B*MX5 zaSYS9F*h1FIXyQ`gteQeITDT~ZH2Ia0~DRiZQvSkb1QGxPPiBr7B13GOV3RYrXpgA zaN;)qBg5_Gkao}D<3lLyuB@WaTfPrByvT$2F1s^0_ z(??ay(#OG4*a|2q0TA;N!68H9)2)DP?(zs0_GRs;RNu4c?3aV0T39%1?CX}3yJXZ0RB3F7;8u? zYY`2o{9m>(J#nC|o12RW2;}MM$?eIIfc!Y(8L111GFE1BHf(zyC>}Kx8<&0wf zTLTo1vP9asxY;3`0e>`_TOi!s#DN%3{~3al3k>!z#m=a|5`{?_$jjUX#KR2+IXV4_ z>mSl6Hx2lIy76zNQCi+EaF7NZg>Xk&V#dRo`EM{LcmKPgKY|! zfLqBU5l(=A^F9YHFuQ ziZsKO!($(s{&v!Q)YJy#>>f&_TlSp4z2{2P*OQfc)HB_e8vbE;ljz}=cT{W&RyZK< z1rd-$8k$QEeHaoc!xYJecRyT)TpdHi@GwLS4@3OJBUk^!`#%HztFoffcm1oz0`i*j z^`E5*VeD^Le~yGd+iB9MPh|;ZklU4s&DJAdVR77HIec{V#E6n?=r+VQa5cLYKnyHV zsQ?LI7JUDWllppqhKRQ0kcy=h{$O9j0kFZp^?ZxbeBUcWJ-3;!fxvy=Wrc@E$biV= z4!~=FZdqVlgMycY2>ptDpNJCu`kT)w6LCW7z!vd?%@v@@1fQb@VW1k*y&rd_{gh@o z+2Vv`FV7uKqvkR(HOkh&Qv>MOFez!VA7s4)^hD;U=I*7&BVFtFJVHHRguF|W|d zpf8ttC3u?JT&~E!%O^h3jqt09Sn5!OzU!pi;f(>OM$Qav@HFY%+YFP91VGI02((s+ z5@V;39}rj8S3qAD6Vuow-P*>Sd^t2#J&3+PJAUt12s#olrCuSw*I>)H$i@O_t5(i^ zGNox>_z;iHikFz67l&$;kr#f}XGU&B8IEj|xOVNF`k2cnO0XmajI6FS!E};=rxoVS2zT4r zM^pS6uIW9O36%k`F0prsFgAg(sEJtjFzPB9x%CR|iU*tg^f0^%jixc#BSV-NRs?R8w5u^PF=2gzE!KY4M;ObCc3d! zp0$;Py-}`LAL_DLRTl2c8^h_QxL>f*N|8h`L=(KCV}TFag5XO%Rd*L$eaVQ`%`NaJ z-;KOr2CwStvl~L;bc&{ zuX1gh6|H5Sst)w*xO|K5*gx}wu7qO`LgTL$hm-_iVL+=_guT|Cd~_vc}o~YiO>v&&fBE5Jkx^X1^oH1UQo&ju|;lH%M9SZ>9`Z zyOvQeTekM*To9g)vsdS@`)9<(-l=pTp7s&p$)pod1xiX@XR^zq2Zx83Q*c|V4^N*f z-QZMzo<*s%U|CHo$c+Ee?E{`l`3ccJR|Oq%LRPG=^i z`4Al&I;$t>Bn~=xXEF7>F`dWLZ@aU`I5O-_e_$6R{-oP+am0F9xvO_;Zwo}(jZ_Z zD@u&j($t7VAatFxu&_Spf#5qbH9FpBBLzNjg5R!L^yPRJOab_%k3eaQw&{oufj3Ho z5l>4cC-hY!y+>U2bgpf90{fpq(iE*j-IZ7qh@EQ}6Ob-0`v(VH?CdAU$59PF+9f&% z=QviY2vQ5i9Q4rC5H2LHO~mI|0$%l%3YU8%@4 zO@y@g1KM(#gPDmOl%n*ABs{JR6!+s3sJcK~p4GZ?(rf@FByd_|@SZmnCdT-97nt6Z zs05b|U^H^k)zy8KU-VQdT!xKe(w|DmOFBSe#=uQvcDOqo0ivt?jn|wWEuS#ivp7O~ zKS}BRs4YE9{FhAcEP!>WB!0J*recjy@77dq?+_soey>%%{GMJq^#ko6g+Ja6+Kp-U za*B{PM?(UVT7w`OF{N@+VSHv4di%4s{9tN<_jy-WjT35;FXcerm-@4qUNTf%WxM%CYB6IP*J6}>V_-)2~P39zk!C+=) zf5q+3*tn+nAvS&Ql25>o@`&E*HCmp6oSn5N-wm2Y8eiB21eB(KYLH=Sab5lrI@fm$B z$!&^*$a(RsTS=Mhu_*h)*LMlpVx0^k8RrKnM36c-Lf4x@b z^`55J+CMW~m{uWd^Ta}jISFv7Z0{Jaff$mplxMiGtD;Ob!nVvA)?9L#Fp8wio+xkf z_2>@>y!hU!dBPxOQSsJ{7QNo8#gipUW04DPhrbpPU@}8eN`NZ39Dv(n5d%>mUg!uRx$9Xx2R}MGDok}8BWxWD}{9E^NEO$ov z_t7<_@Mt`F>KC-2-&Zv?RIl{^srxzPiQcep| zDL%h9g#9mkZ_lb&pQ7B5X~MpxX!L_dvb&B9p2tcJU=7A2}+79VX_w5#J6lHhLEpYMVtt&H^AoNC`vd6i;`%E7<^4fEy z_A+Z(SEdJbBVHA#Dl01=9v+%)M@7AEzBv(%n$I{6NGyvuipb`yu)EvXEQqR!h*<9z z^_)|wGoSSOWxbRn$?d*5MM}R^#r~jVx((bi0?A8ix!u%!ip80Z#zxU|Q_-9LZq#3B zHQklOg7l?{f|k{Nwn7s>>Q~kq?JjR`ClvJer#?1b|1uPtDN>+OWQEl44Huv`*_|cL zi-dFN4DJZiPwcL7&K(YS$PxxrpDZWihAh%cVji1xfoX>)Ra>*I3eN+}h1OjmIesT2 zoh|s0=OkPQ5Bo<_dpo8|Y3fGOxAdSwM;Y-0eNL=$czOtv$JzEz3wY8ls*w&CLyQYO z&VIPlnCFExKQ&Y1bkm-$LbkbgVnT$7 zlYT(cNK43X<>S;kG~&cMfyM;uz(d3gvc9&~6Q_UHmD8HcmB8BK|9eIMEUjSy_w3-? z-a5Tpqvq3CTq}1ZC(pLmrM1OzvNlIDwpvzJ7K5mJjV}X$g@}yw=;YwqMV-HHVXH3h z^QWrkn|3HF=)K4H?J9M>&XgBEGg-5-v)8=1%#iNU{OBQf;m!ee>-+y5}x;gs7jDm46{HKY2k7@!hocD|Xqv8xM>9H|7pw0g_G^zBN>t ztza2@>(4L+hY+*y&wJh1#kE0!gqH>kF|7xDe0;3Y{^H_0xrH7AB#s_E-|lghF=@{? zp$KP`#U$n@+i%*jA551R-U}igN<6aZAL&CZn7~!D?`wO4T;U2>p1el5~4GYuSULe1FB})g zaLpyWx!X3P^-~J-#>(DJlU02n?9($R-tet$Is}F(n1>B34+|qC{MXmKN@F0-4pMI zn^t--_=5PrLdbZA_`g-hLdCz0#bZJZySm4p;KH2#gCs)S-7$?vq zNs%iV!;1W#Q1wbX zwrD3;HurDrXCqamdst8L2wqCtD7=Sb{x4!6)~E?^n8jP6oy9im;4_<xAV|IHd#~D>!Ox;X!g#@JH?%Qnoffl7Dz+DN ze#h-y6_)ZQEsV9hmoW#m@dS;B_U`8N#bixSSXqo_XINFzK6_7>;zq<>*LfSJFAR(j zvYWXrmtLuxaEBATVef|X0#r&gr1SOWT}SMbCLKdIAS|3c7E0MhmTB#Y%~${eA(F^~ z$VUo81Ww`~E8`;7 zN}ypzxinN%ejMhwOM9u|l16nLHatV`Uq|x_7mnbVXoU|2tJPZy1^7Q4`l!~XFP4vb zFmGBg{0)ZSr)Fyzt}?V8*%(g&q1U4B93I6d?7Hm^l{6Wh?D`;NK{}AuH6ZxH! z-Pv)Il8DzG3qUdB=JV;dp}MzqZf6sPB7Va*jgm(*shWZaR)c4n5)#egKF7`OeD^8X z!&OX7$ur(zC1(++*53x{ZgqgI6% z;h$?mPKUr5Qu$)wPugeh%(uU-mDBEA*gEqIP@D_B|GHG)HUgcc>_q%BHWm z+~V8Tye`iSVozo-9=|5J!(=1kz4UJOB(Ab@_sLVZ4cz(6Re3f^I;nJW#H9-MCJ>ig z-|ZsZJV0ymy*`gcrU;NK2!+Z|Zhd{8$oxUD#M0%F(kl8wj$hoAfGkl7Kg^bVRR zJNbj>{LR9`*Po=`)qw#frLa(A(HhXX*TLK60?r?fbb}_`pInegiNM{S27D3loy%Zqn))M!%Q-^VvoDkN~DO=$PI z0fbn5iFk2uSs)79q;5PNnN^@O4hcM`yqiB*XKuSLPxoygS^0em_tj13^Qi9@gzq&9 z$39A$Jj5LoUx|+4PQq^t5O=ehh&1U-08~zlB<+P{8k>xtMi=)167U{&u`! zXW^*8{!evFzw4``Lu6|Pl#H&?sas8I>)Xb;vgG-G1k8}iuYA8X^dVVFgXd{!7~MhK z&~TB3K+;xUiQ&EpuwrBDaS9V-^)ydT5$|5EyDeQ>Muyfb!C`#u)0n-9dcFz7uaoPG zdlL<}Z#r!qsTi-=#G8CXXS5u~{hBPNlQXAl`{HiCeyK^y%zQs)oAcDgb!dojYK3E9 zVXOp7kN^i$AE8jKOA?nWN_WJ>-V46Xd*u79*D`@wwf>87&iWubE<^qHS>xwXs=X{& zS3VsEd(KsEECugWPsaH`4Sxr>X-gt8NZpwrCf%^!3A zmm#f|C@+1QO|#0BG6N^^O`Oi-z-D9A&6U1H9N`DD5W9go;sjRJGz($r@(|N96MWaA z+S+^jxE`55)@7-kgsazyfg8P{4jfRcPi*fUxGL|jslSJErC)BdV1(M z{*5Ywau|sW zB<sLrQOn$U*r(05c8x~Bky)V&UB!z| z#_dAZL>7F%Q5q*zTsCav4ESodgn>aaNt7Xdh1b9_Igj;)>;+l!Y?)K#Rn1dlMWz0N z5J?ZzSo+|zZ0W%b;_wgd4*f8+@9z^-d#((5wh3$e6+6@QFse3jM2s=Ppnk$d2Bx8w zt;d*4a_W9<;ZhJ>BCdB9!9o6=@~sAKol;j`95D&PN2yah>3Iy>mblx~J-4 z=_~{4M4&UyVlC!Kwvi+9wcP*rt(pJ6m-DZEo@3A9TVgO40rQG>>#xGmue0 zoka|8*}aLV&EH2$T;0$~?RyyHo-+KUQDc0znwE85?%ac)(+4GV}=^vYJ3?q|wtDsO^D7GC1_6%ful4ja) z9kA1OzIgb)I262;8)W87gby!STAx4Ll*Xj|I3m85y6!j2ot2V64u$)4FWFsLZqjLS zC;o%eDx5xwo3nK_Q{hvWXgz=^m=mbW+Wfe)cIpp)V&SUh+LT}#scFL%yrsGpE=3G! zP)*}q@Ff+_ep46uLfj?Y( z3G25y?2U4IiEZ!t!=BsYRVifqB1==_HeM}&JD|l2`*#<$(J==l_I-i29C7#iwWJDI z?xf7D$*{&w+Zi3+(jqrGIqA7I60$enbhukRJXT$B>qN~~WLXAtLWRN+3M=QP=b}@$ zIksjd)r`DX4Okx$qqz?hjYEf)0U2?E=ky2B=!Js@N2Vi5*F6{-(W3ex<%A5k?J46j z*U(;R-E&4j21gl~2;2KkRaljnw}9VvbSS#JyMK&{SuYR1;!8_QJG;E3XJyrNb#)Cb zQ-H{8suJkzx2wXT1(+LR@*&42s1>oSgImT{g~ z@KE{%&Rz^WH$Tq=4T2zBj#H}UW@f-}INa3CYhnlFTmlQqVH~xiU!R>dgc5!BI z$|MFw@r8Gh;ZXEGB+mBIHxRGz^D8Iu`#Lb;lamm&e@T0wr!SqEz_10_5{`U~l<_mnWufM;D=hmy;-CagGf3ewm zq~t~Rz`(#-lwhkRu>7;Lv!(5AwR)HNRAFl}VPRqMIxTf|>&}mNZ;_MJ-@6wU7k4`% zBH|tcL$%!~I|Kr$sH%Dm+|4T};JbWdYkSzsowc;N+4G4_$<^JxV{tL-O@E58+lq25 zv-InVK;Tip@|D%qAK$cde;*xj^6-3I??)%1(FOH{(OeZuT3RkCDMlz1dU3R^huQs@?^6>I51B>+BHQa}Mevxan_A_4Nvl!*K zEh((uVZOOEQ18heq+uUEjORS3dj0zK)=X`26@;5xg*VHfc>Tw1Xy@FVDeo+xA(NQh zr*_AyrO?}Fe-4M4Bwd(+*z`}v?Zl`b$!B=&OvcB@x4$JJRfWNHCXe^#(@99lczXP9 zu1?i7HMt%?4lgySr(#`VFk80x<{%Ed(A8f-f7e*wH2*B#51z{pVom zH<(3hU|LmGr0hfLea63(C7Jz`nqz=DE;eSLktSI0PB z;SqZ3PXG$_^pXzNNrWuFg8t+OvX6R)>&CwD=91AdNn$|yt!%X zx~PeH@}3H&t{xE=NB*Q^{HrS>H`R0?p=Or4^Y(-=%hz98TAHjG z4>K|{>g?>S0PsnMp1&h0DfuFfwRFx%M^~4Uii+du(`c9ZMq%3_=6sE`oLgej(niy` z-rn9NVZK@?znjH+ho0`$A6x*uf89|@jCLN!UT!uGF;pR}&P-!JFezjRXHL`M zmGu~M2UJ;c43^1khVf=nynA@Ke2B6o8F)Q(2v-a*_b0-WH48Y{*h0R2~tlK+041c!>U_<_Fi8LWGF(N98LrhF>+GVvn=53IQ(lOgxd{OK<)6Re&bDDB( zMS3-x%l{*6Zpsjv{Sanx2fhDwG^F59y5ELHUTfBURO)NsqJU6BPF?j6Q?(b zKg&Np_yGyv82vaoIr;A0yV*6B&)RSmBCg2*Rf|r2)sLm4WA1{%eZ$ZjQ)|DVAo=Id zpZ7)IdCRMtFXS{O1WMsILAaW)rHR@~d2Zb?FfdSl@#2Z7RFP=yG{dnK-Lk`M$NNpp z!<)>LW`{$HVFY8=8w-oJ?Xj{2VyvKm00N*~iE%B=$a`yq4d95Q228wrXegFN)_bJf z0tzxHHGV#vDVGKfmIvm7Rl(qMDj;W>p+Onqj*~pA@bLmIcx+|Rl|N9t9dHZOb2~d8 zfN5(MHy7xtpq^Nk&hBpf@lSoiGpwiD4V#YA!NCEed|-n){1Q+iRT9bC+S=-0tv)a~ zcz(HfGXxAN1MQ*YX_ClHbJa47wjp3T2uV+WNKVT<5}@;?rbe&a={?zZFB!~2-bzi) zd+4)bu0+3q;OXq~W}iGaV(25n!sq*J3ipyz+}6fkZV;f)he)sQCuQZ+vM8* z0q7Lhf*J1J>jawrj@JS(KuEaE{hXg`A8`#A8E)OC$A*WG?H{QhO*c+B_@t z_X`LR1v~@de3zES6nrtBWmsiH`S;&{M>9(&01#$oW(x=M$2+-1sw|(-I!HBf%Y>npLA|un$ z(*ryt(=E_^GgD*l@7&eUAO#G+1(M2EE-pilu78u8qmK2ztO#89@s)3(*z|_f<|?d| zTCPhc5F*Qs7yV|9rGNId8A#vy1VRkOHl|g~sB4V*>ye8F=B~O~1VFp?I$Y?JCr=b} zM|pZCw>@DKxc2t!?m#7L7z_r~Q&(RftagY&*Ggphuhgo2PFc8mr63SSw?qQ3 z654L5i%};21AnVx!sXZatHnw@x3O`1Fo$un6vc1y@H9J%;YWbC-LF-0CE8_6g3ti*AHbhg949m?;Pj{kypW=ngR8^oS=t^!z zVp%(U#CB}CC@CrB2QgTxJ$sitJq-WaYN%)U*wX!`%rHHD_oWGhjWxU5r_pH5Cb(RQs+FiRB_ZUeG53o1tLpK-)X*qdL%10MT4Q z*oPFfh|4>({PuF}0%^fzYW=GUrYXjc4BxAN>s zF=CI<5?jrF{r>$6K0n<1KKGn^&i&<{*Ez2k3v)xJTfDa@C@7eWjr6Rp6MtP1bTrrf zjhvn0bx;Ob8R}4Ahxj(H2kIBvX4(`Kl?e%5$Z)eiO9#8Ug|CR53 z##1FG{0yLf6HwY9v~xIN;uH_@`DOKM%xh9e>O%wP90eMwRy0#GX^W0L5=YwmCNMyN zWqJC0baX~{qN1o11^b2NjdA2b5Pi~#sCMIk-d|i&Mrz2xPBWQEz9L`lWT_2vFw!(> z(@n31ZVL%M1L_8_xzQnGS#>QOqqyHK-|1lG1PZ>-X`qv(?$rMvC=S+P)c)UES)!+j zBpxr&FQJBKEb()wBc?36SFhZC;dE)5(l@r9pg#2Zxhip}w=z~35I1C1_ccNyP`;Vl zcDa4mO2B~0OxHDqmOmjYm%(ZOlcdu7G9kB$%ecUh%qNPQcK^A`Gl!Tsh8RaVFsnsYdx#$S_jk( zw3f{qh`#KDD~GZOjA8uP&DuJX5qqjyQBj=5jLjUr|IVaOutOe%@z*xgSv5`|S^5_e zyONe#@1CE_E_F9@SNrq6Nl&1X+W&kr6;2O>s;bE$S8Z(4->+5ol^d*gQB_dOY@)+F zCzn-3@(;Dls4ousk_B*htJnK;;gHX<_mV+ibBPojbXA5*;3ICK3E8 zYeRQ?&ZKl@Gd4J*og0y?jvGHPb`G~Ibe|rOGyPVXDc8hY0cN5YD?XYvuvCD2IbUT% zJmgD_&F?fY_vo+^Aa!M55#!EXe02_NH+gF2pmMd9SLJK=%@~wp;l{YMc-w0BVO{%9 zp}=!l)1SwOo9s1hle>_BFl>akc{boyJu$^}2;gBnXDG{|O>Qm@^GG)Qq`jZ$H0lsN{qJ@zAktAFDm`2~&}jKAGw!C|BpaqwPFVW_-qM#Ux+4oUOK7d*{VPy)q-dU>ht5)yr1 z=|$=H&#*x`ZN4_RRrag?{llk@IAT?oujD+yinr$Dq*~#rj9tkO`q~L-#Iq0K0#Ih5 zksYgJKjFqsk)J=Xj9ZR{3d6BN(U_(0-Fg@^O_bbs}h_HDoBM_vE4!xn!9=*ZWGY z*tu2{Dy3l=!`qr}SB}d2Ma^j(z_`eCVzRJi!mWHM9M!mjuIj2&K}0 z{`|rTUIfqM4U|nZh|OAXBC@Fn+!k(bSPuB(?1?t6;TQub)_)%?>D^hDsYNs2H8;W$ z__BEWMW(l68U(E)t9Ob!bu0X;HSPK77@+}AvT+S4BYqahT-ZZe1yFVd&VS{Y^8O}O zJ+p-#5E>f_gj5@nCH=Bz`3A9Pr2^RYTx+GpFA+3lEM1dX4{d2`8emPIekE2+d`HO4 zN$Y*^2B`*IS~~=+yc3;UWG%dMm%KW*ZAq8>c(pQV*>5J}mam?bG@Yp=Dp{4YB+6ZW z29ut`N6W0m403B0^S&TDMW2#u+`QSt?ovS<1rT>N8yvEbg2io}VY#zC_AP#2&Gh-@jq!)EWX>Q~#$8fk8ishGn9 z<&SY#8ZrXS#~@%V4K6yHY;edXB3n{E9&o^5v9V~B`h(7I4lgT5|A|evaKUeS&xS<)#S(wsN)u0Xa9r|RJtR!< zG6p+QMs+jrF@@F2VNPR;A zDC6vq%Tdi=vTMO1K~@Q$=hYUb-l#XHO-$a7YM|v9l4u#pLx7(X3Xq`JZIp=2&>R7= zxZnSDU9}{XxCO|c%k7-dPofAU7KRRkgZky7I+R3(bAq9ZZy+#*Kx#`mc=@z59_)Hg zC5`_fZ)zlrQ50QbFB_8%}`@;6dC`$HbFuHeg&vv89M z0|wCEI|3;JK5Y~gFrbYXxU_luPbDV{(jx4S5Elkn1Cgg;w`TI6(Ww6R=A!joS; z|Dc&Vac&{*W98vdG(O7RYt@5A%Pzfm?{pEaUtHIFs=jW{eZaLV2z*>>eKmmWuH;KbkqI z+Ww`5hPwGacFrMJ_OwhJdaM)%-P#?ByY)jIL*0!ss<)-ef+kw}zQ8R);RmzZx^Onz zM#T2@iR+)hx?EhczR_%Al=QG_I!joymo4S8~_GSBKqw+Hr0jqQ9#-nB; zvU1#8_EEf~VpLi3Qs~#zz2E2MaPK2?I>+YL@cOy=A>j?#HLsv2AF69Bwz- zn6#70p}YSK8Vip5AY(T{t^b&EZXYHz)Eq7}lp?dA?!^~=VE1P*GT8obNiihVVm(wS z+QTi*kJ|k9HAL8HeI2~CuW7`i5sy|G(#q9yA?CCpw;unOpSxjF5dS``W*#rtN*^pH zGT7#Aat6~k(+z+F(Vj*>;~=_$wDGSGF8LWHS>M9PEiE)MkHL?*&UbsuOdhX_T}^W! zXdTkqv@9Tz)*pMfuwnWs>th&csQ?K>9p9ja;n&9AnTKH&l*BJ9T6U?2RjPZhOvGG+ zwBngb`5};5L|9P_Xr_(u;v0*nMp%f`;>#aX%o&HLS^f)Q2bI>ml08A&$dKfM9`T@c z*C)GMwk0I?pZvA#hx2RQItf<4@dDi>jUG!oH2Hf2cmMNG?%9ddEKd~aBUZ3}mZO)@ zN@$!S%9^FOMznW<3qF~i0^rK;0MnLk@6%$Q&b9Iw#aK`D&9Kfp^S~UlCTL!*YHkv6 zWhd%ILevA{O`UyeQ)#1BNB0R2_|javeTr-zR_{hCde8G@fce9uX_?HbZ8!5spy+Hw z#pK@n5kG8{#Q7Rggt^4qQtz#;TU69og0e&vX8`0fi-2DzRhs7|9ON=)x4}Zz5|GYU zhNw{jZ; f|CErfQWf%o9kh9p1%7(>cv?e literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/pinterest/index.html b/content/ko/case-studies/pinterest/index.html new file mode 100644 index 0000000000..56bf0ecbc5 --- /dev/null +++ b/content/ko/case-studies/pinterest/index.html @@ -0,0 +1,109 @@ +--- +title: Pinterest Case Study +linkTitle: Pinterest +case_study_styles: true +cid: caseStudies +css: /css/style_case_studies.css +featured: false +weight: 30 +quote: > + We are in the position to run things at scale, in a public cloud environment, and test things out in way that a lot of people might not be able to do. +--- + + +
+

CASE STUDY:
Pinning Its Past, Present, and Future on Cloud Native + +

+ +
+ +
+ Company  Pinterest     Location  San Francisco, California     Industry  Web and Mobile App +
+ +
+
+
+
+

Challenge

+ After eight years in existence, Pinterest had grown into 1,000 microservices and multiple layers of infrastructure and diverse set-up tools and platforms. In 2016 the company launched a roadmap towards a new compute platform, led by the vision of creating the fastest path from an idea to production, without making engineers worry about the underlying infrastructure. + +
+ +

Solution

+ The first phase involved moving services to Docker containers. Once these services went into production in early 2017, the team began looking at orchestration to help create efficiencies and manage them in a decentralized way. After an evaluation of various solutions, Pinterest went with Kubernetes. + +
+ +
+ +

Impact

+ "By moving to Kubernetes the team was able to build on-demand scaling and new failover policies, in addition to simplifying the overall deployment and management of a complicated piece of infrastructure such as Jenkins," says Micheal Benedict, Product Manager for the Cloud and the Data Infrastructure Group at Pinterest. "We not only saw reduced build times but also huge efficiency wins. For instance, the team reclaimed over 80 percent of capacity during non-peak hours. As a result, the Jenkins Kubernetes cluster now uses 30 percent less instance-hours per-day when compared to the previous static cluster." + + +
+ +
+
+
+
+"So far it’s been good, especially the elasticity around how we can configure our Jenkins workloads on that Kubernetes shared cluster. That is the win we were pushing for."

— Micheal Benedict, Product Manager for the Cloud and the Data Infrastructure Group at Pinterest + +
+
+
+
+

Pinterest was born on the cloud—running on AWS since day one in 2010—but even cloud native companies can experience some growing pains. Since its launch, Pinterest has become a household name, with more than 200 million active monthly users and 100 billion objects saved. Underneath the hood, there are 1,000 microservices running and hundreds of thousands of data jobs.

+With such growth came layers of infrastructure and diverse set-up tools and platforms for the different workloads, resulting in an inconsistent and complex end-to-end developer experience, and ultimately less velocity to get to production. +So in 2016, the company launched a roadmap toward a new compute platform, led by the vision of having the fastest path from an idea to production, without making engineers worry about the underlying infrastructure.

+The first phase involved moving to Docker. "Pinterest has been heavily running on virtual machines, on EC2 instances directly, for the longest time," says Micheal Benedict, Product Manager for the Cloud and the Data Infrastructure Group. "To solve the problem around packaging software and not make engineers own portions of the fleet and those kinds of challenges, we standardized the packaging mechanism and then moved that to the container on top of the VM. Not many drastic changes. We didn’t want to boil the ocean at that point." + +
+
+
+
+ "Though Kubernetes lacked certain things we wanted, we realized that by the time we get to productionizing many of those things, we’ll be able to leverage what the community is doing."

— MICHEAL BENEDICT, PRODUCT MANAGER FOR THE CLOUD AND THE DATA INFRASTRUCTURE GROUP AT PINTEREST +
+
+
+
+ +The first service that was migrated was the monolith API fleet that powers most of Pinterest. At the same time, Benedict’s infrastructure governance team built chargeback and capacity planning systems to analyze how the company uses its virtual machines on AWS. "It became clear that running on VMs is just not sustainable with what we’re doing," says Benedict. "A lot of resources were underutilized. There were efficiency efforts, which worked fine at a certain scale, but now you have to move to a more decentralized way of managing that. So orchestration was something we thought could help solve that piece."

+That led to the second phase of the roadmap. In July 2017, after an eight-week evaluation period, the team chose Kubernetes over other orchestration platforms. "Kubernetes lacked certain things at the time—for example, we wanted Spark on Kubernetes," says Benedict. "But we realized that the dev cycles we would put in to even try building that is well worth the outcome, both for Pinterest as well as the community. We’ve been in those conversations in the Big Data SIG. We realized that by the time we get to productionizing many of those things, we’ll be able to leverage what the community is doing."

+At the beginning of 2018, the team began onboarding its first use case into the Kubernetes system: Jenkins workloads. "Although we have builds happening during a certain period of the day, we always need to allocate peak capacity," says Benedict. "They don’t have any auto-scaling capabilities, so that capacity stays constant. It is difficult to speed up builds because ramping up takes more time. So given those kind of concerns, we thought that would be a perfect use case for us to work on." + + +
+
+
+
+"So far it’s been good, especially the elasticity around how we can configure our Jenkins workloads on Kubernetes shared cluster. That is the win we were pushing for."

— MICHEAL BENEDICT, PRODUCT MANAGER FOR THE CLOUD AND THE DATA INFRASTRUCTURE GROUP AT PINTEREST +
+
+ +
+
+ They ramped up the cluster, and working with a team of four people, got the Jenkins Kubernetes cluster ready for production. "We still have our static Jenkins cluster," says Benedict, "but on Kubernetes, we are doing similar builds, testing the entire pipeline, getting the artifact ready and just doing the comparison to see, how much time did it take to build over here. Is the SLA okay, is the artifact generated correct, are there issues there?"

+"So far it’s been good," he adds, "especially the elasticity around how we can configure our Jenkins workloads on Kubernetes shared cluster. That is the win we were pushing for."

+By the end of Q1 2018, the team successfully migrated Jenkins Master to run natively on Kubernetes and also collaborated on the Jenkins Kubernetes Plugin to manage the lifecycle of workers. "We’re currently building the entire Pinterest JVM stack (one of the larger monorepos at Pinterest which was recently bazelized) on this new cluster," says Benedict. "At peak, we run thousands of pods on a few hundred nodes. Overall, by moving to Kubernetes the team was able to build on-demand scaling and new failover policies, in addition to simplifying the overall deployment and management of a complicated piece of infrastructure such as Jenkins. We not only saw reduced build times but also huge efficiency wins. For instance, the team reclaimed over 80 percent of capacity during non-peak hours. As a result, the Jenkins Kubernetes cluster now uses 30 percent less instance-hours per-day when compared to the previous static cluster." + + +
+ +
+
+ "We are in the position to run things at scale, in a public cloud environment, and test things out in way that a lot of people might not be able to do."

— MICHEAL BENEDICT, PRODUCT MANAGER FOR THE CLOUD AND THE DATA INFRASTRUCTURE GROUP AT PINTEREST +
+
+ +
+ Benedict points to a "pretty robust roadmap" going forward. In addition to the Pinterest big data team’s experiments with Spark on Kubernetes, the company collaborated with Amazon’s EKS team on an ENI/CNI plug in.

+Once the Jenkins cluster is up and running out of dark mode, Benedict hopes to establish best practices, including having governance primitives established—including integration with the chargeback system—before moving on to migrating the next service. "We have a healthy pipeline of use-cases to be on-boarded. After Jenkins, we want to enable support for Tensorflow and Apache Spark. At some point, we aim to move the company’s monolithic API service. If we move that and understand the complexity around that, it builds our confidence," says Benedict. "It sets us up for migration of all our other services."

+After years of being a cloud native pioneer, Pinterest is eager to share its ongoing journey. "We are in the position to run things at scale, in a public cloud environment, and test things out in way that a lot of people might not be able to do," says Benedict. "We’re in a great position to contribute back some of those learnings." + + + +
+ +
diff --git a/content/ko/case-studies/pinterest/pinterest_feature.png b/content/ko/case-studies/pinterest/pinterest_feature.png new file mode 100644 index 0000000000000000000000000000000000000000..ea5d6257894681f6afb0f969c3d0cb5b4bd371ac GIT binary patch literal 9118 zcmch7RahL`wl(gW;BE;JyphJeAwX~l79dCijng!q#yxlv2=4AKL4yXD;2PWs?r_OD zd!PS**!Mo%hg;vOnyXgLF~`)m3R6>&$Hk(+LO?*kRd_9{@g!%SZXXP!rzes@^8S+` zhk=H|xf#>?pdwdCRv5)$I#25l&K*gL~a?9IS`b3>lE|H2CX zk62+DDA)w%0M&AEu>HFW)T|s}4$f8%j&w3If2z*U$xp|kY+?qn|8vaxr&RxrTow$4 zxPr~)pbmC)|H`#6W7=(aQKv|$Pw=}N|5Xj3f$M6@{{2%i7 z7nb+G#d1A4!}X_8{=dfg`{=0~|7`y=22U^l86>d%)AT@}#v~Cf8W{nB_Pc_tl$QJ4 zz9IU1BJG=Wpo^tjH6({%9K%l%2}2es6zdhq{!Hbue3y(EyPIysv}W`v5%EB3Un(lQ zudZ5*%tj0al9Zw^QB{nzfQwtTNVUku{c`dw`?t&&8QVLzLaV}qt%)8-m#s&a@a1GZ z0t^%uB!6Q((UEKk$T%TN8yW$_vp?(qRQw|u73#i%V8~_yU$oAr7g*9grY5HQ5kqo_ zlAh@+mdvKNhggr%BJ<%uulhqPy_TpcBCN3(|30)mzE6&GpwE3?B(WYBtW`jByW z2HV_x(=V7uVM3#Io`xGr4(>YMA$=fZyb94F1oKA%bQu8xLXChT5|pMqjZuMS?zL#m zI4;42NavX2E9&oem^JB9p#)Q04_UjJmI0HKqL!Ju<~A}50UD%Up;#_;;yvh^wM4S z4Qu^d#;!wNsy2>{#28Vwk4+t*R1I^7=pR6m!9-ND;E3n+tCi{xSEscjBdguI`|m3y zrs#m*%w$U>V@{1vBsp(sMurkXf})x{W+`V(^!Q6ZL3dvA!>l zZ@FZ;)>N0Z#2({pI(k#c$`P#syD~>nRtm`IREiMwvbE+}^Jh+PFvK+d7O}`F=u|vK zq37Plv-X8qabOO<-mlVxNh!P$uNdd%NZ@tZ@*g_bxKaXJV=kuOieBYt>Df`oOkXz2 z2mLhv9{m`3raaNF?PcGl~;xtWFh;9KUTgV=ljG#yer(y!myydaaN&hhyGqOp$RyNgfz*p%xSAq%nP0YDs}C@k&ZI{W-~Ph-^1?Ja2X<+h$Y{5 z?DkYM@jUPITv2VYZ$)!7RP4H9yRdItQynilkFYurRZ{!LS}D_d*ELl>GDgYTvV*tW z`;qmnp#N)B0&BGi|IeE&eMe56+AT*(Bzb1nGOE<>#~fB`gj0;1E>q8J={KD-N}LmEmbxqUz2tvgA4z;+VS%+O*yHwdcqj}wj_D?e6@XPH;ZW5Yt z{|=54>?d$ld9bOZ5$y<8tiNKT^P`L^5Z(fv$v>*(ajmI_s}esesj=60PRc_qY`T#} z`*FTAp6d1n%#zINwkDq@1e>&M<6-br^j_e$kB}6KP?=~fDVZ1uS7mD0uM@!kIaSZ` zh|lpm?<;R%Xmj<;f#d0S0_J0F_+WYfsA&GVt(7e04FdmtOmRl_VM@~J86!;b!)HxY zqE1W>WB9?%hgqIg9A0wBXM$tq4EaDQt9%t1M!hgRGb_28fnuO-%LPe)pMX#>j+XWd zc5Iko6@H&ctd!}2Yg#ynhl*L)y@#IZ{OaIZv&WxoWknX2jAs-&NukfO8~DB7{<+f# zqmQ~ZMb;$Ms5Xd7tmOTntz3<4WnG!~$l6F5t9mqup|qG`D&9 z6S`ab9B|n7Q9&gXRuIaH#5UD>;i|)5-XUrjBK3nbrnzT|Q%SsSp$zp)K)oGt3tO9W zs}Me1rOobH`n196fZ7&<+rrkWox&Nv#X%w>j7NrU10GHaNfY}$B_ejzT78+S$uK|CaBgv# z6lq866Kt9gg_VgXG-Ygfz_~;v@?nMt5}M^=I5FFGHHc1wmPI`aY>xJq%CFv;$W`n% zdRrmc83^@nd5Co5bx3AnVkW+?_;6%w`OSy@BegIKx-wewIm&aXvAmmlEV5%}!9rQv zd;7@U5;IxK47@2dqS*xgd+#!NX>&eJz(Y}cPiYMc;O%->>``zNq1Sr=fGAuErekow zKfUp4%a=4gTokn2CfWoU((m6mY*6|T;;9^PI_ZPweHm$XfkP4!`hjQ4+fB9e(+i{% zAWIZG!HMpxDeD=n6gf~#diRGaK( zSZ62f!xrk#Zwxk84@y{eU(+a7S>Rv>tS5F`$I0}zAf5Hp-Ao%rQwmpD%KFP~C&sxy zc$C#Iqx%QtNs~ii^BOyNd>5$zE_rhO?)hOMD)dKMjrglT#1YYV%@&N*l9?sy7+D!L z#VJY=HaFI&HnoUJa>yC|8sng?iL6|cUt2iOq_>eq?Y8y%13!*xXNQsq zR&H4^r-=w`Bn>DtzB(vAwtr=JZT%u36*;N{68|M?{-KIW8oM+Hqezv!{wJ-I8SB^BcwbPJGBlh;#MUZ&uOQR+#)#Ff=DGcDg`ZHbYN>(+h$VJd>I zZoG^U5~Z{9#AJpdtJ79``r{3|l4tpET>Ca}{k1H7#GvkIMUU{erLGZFM#qE86iaYp zmnJ?%;q2CMrBL9;MA=D6Q(OLM`8c{{74PeEf}mKTx{P@CSXT zLo*f<#}vrY=-DJcvE943*86&v`0hJ;>u+ER4+5ND@$-gX zy5%J9EaXrf*_$!ZADQq&F%HJt{nFc&wZ7+iRRB{gw5+=9>{p{_opWVD#@x_zauoDj z9K5BL13o)_uo>W#%7&1XWZxL$eY+Y-5=lf*u9g+cm@=U69vf%q zabLY!!(c>Khob_V$H60mM$NQ$7@kp$MV*6^Q?ug}iaGC$EH*5GZAyyU`2v!|qnl*2 z+$^T=!~L#2O)VO0NUz-sXutCvyxZ^=LrO3|$io*Ma~X2peC?NXU&CV@2`LFfKG`;V z#Y-_7K6lq4od$*x7ioNZHGn{B`;}nQgTBJ0ceD6qHXZBnoQ;`0v2=?JMOHFcS3Dn> zkZMLfN_tQLe_U&tbWD2YLZEIv)rkaPOMIf6ag{xn1GiCSOL28rZ1FlKw z53JSbY2`)U4Wr48j%$r~T_IMAR^rQfr)9q9r8R^xnPHhI=A7Bg5_IS@AP^QArZI(p zJ0(_3;#_0<@{oLd!wbl-dSi>ndq`#m?XXnF1t{6)KG<$r zS!yxy!)e!7p6m#kb{{ZSX`l`B*;gq1Bxf(G+-Mp1c~MPWW*Pp7i|q4t1=A6-A2W8p z%=F(nv@01R3dECta@bD!@C?-MXu%@Hi7huLTD5ZbyX|u&u5a8mhUVU*YaM)4|`w*y`N*&M9-7 zh2Fa$_x6AsfHg1KnVt%z&{b`l?kUs{ExFCO#cmg8bRjfQ5R0e=j=OOwrp@h8$79Bi z&6U+u61)xjjMbPwXPN!mmL#MdPec-GeIsR z+XWKcXi}L0b;wI8qI!890D>tGd-S%-mS(a=YEZ-Y4#ZjqZH@+ zNm;Kt2RlS@(fQ&b#y5m+FsG*3H=P_*9<@MnK@rh8JZG!O-Ei#BrfrGKs1K7#t8RqL zu^ZCCL>Xf}f4IAvNdM@bMr8G67e0wQRL%fGRk)cDA_HtCrx3_7H@cXL%e==HEt zT$05)yw4Ijo_JM|;cOxj(xi?DA_~?S8&m=m6r`^twb&C7GBn9r^bHal zrjoouL6*ZpmG%?kGhT5!M4qugB%8!A|2mA`nK31_hsbhtks87oAxLF|5C?#Qzzs-Y zsl2A)pq#-dnX_q_c#F&YQzb;xULB@59==^_{n+sk-V>#W*AqbluHsHvT`QR?fqbfm*DJQS zrklFkbmkjdEb(jaCx(}Ct%su6P?u3}SAv4#Ytm|*3dM$nPOT;bqFB*?VQxJ8_Nnz( zL!UEZCmsp~1*0KH@~22N6ksu{I_Hd5On0vBb_Bg9x?`|vkaNUX2K+2o+ zlZh@d7LrTD8@70GxODzypVuw8b)XxljQ`TuTPi=*$xLqVv+Gy(i`b%UVy{Bq^s1RR z1Ce*-8b-C3xk8~Ggvj4z%4q@gw7rhaUX%K}OOU=%e~e;Ut$U&9;m{H~zS70^@#ZKo z96(1+yMK|+QkQy(li?`F%ejGmhzf=wv&^^Y^3umkbnXIct(eS+oh}53jo$T8eRjFt z$<6G|S(FG|_O9Spym*Ux*R@Sy33u37(EBc#;EVeT6$oVGnGBxrud~=&7GhR|ey+|8 zq`(5QBFGLF(aaiVp}ejJ?upnZ5isIpo<+*!M{(M%3O8+x&e7d+6%m(rTJh9oy=U+T zr{2*H!#1~{@iWzymQC_vTQ24XxZm+I(N>78`-mOcbsl>}aW#dse0W}M^zG>-M#aYt zo|hO+8#U+J9rXML;3?{qfIzN{XP8?TAHKb!8yIDQwetvX8EK9;?Y5b8A`a~+RQ7QO zzajIb0)0~&fYLS>@6NVC4!qGkaIn~D=U(a4DQQo8uw6E`={J(%vgFS8M*K*>Pk~Fe zrH6$sVmLMg>XMltNERHupT9(}es$q5D!@h^QX0N`?PR$=ox6L#veq;h;M=_yUPgK!Fr6{!PoKC!TCKD?^{dYU1}xT{)b}G`-J!cH|!Kj+(7*2$pHSl!w=;TWnw|M`&Oq+ZA;jnt%}-9+dtDAr5#g~TrRYsA=aDgiMT zIp%;Us}?Q(<8URkyS*;~C)WhAjv8YfbmHQiqNhU}(ifFxusf(9Bet=jCX)4j!S|VJ z`5T}EW9J*mG~pnZf~?Ill~_|s^m9zAt)+^w`g}mM zD?j`Mcf-d!5N|h?MDtqFJ124YR|`o@T#q3{A^OB~vU~`{%k!=b7nf4&)3A7pl8gJ# zTN2J!p|`ol4yH}7ZQgPbF$K~LN%3fKtNk8UGW=QCH2>UN22t>*z(kFmnvaKOq0_Wv z!o65oL&4*6dJui&%&|7#3x)%hq;!@VfX3&#{RnUT#d*sf+7=Nvj{&wXH*&L$<>w*i zOG6ST6mf(pCZ~RfC8Y)pOj}DL(FGDJNQ4{^NS#S%&4JhZD_r`Xam3Q8+S3aOeB;U) z7>U}nm2y2XZgzG;U}2dB+9B^-(%a?)DzuRsCm#1u67pTt9Iv_>G=( z)wlQX!53jSTuP#vBT~HOnEchIJ(oX%U&A57?;X*|f>xKR#KH_28Nv~ym`cxwfn>lF zt?7kJ!FR}kzQjqu&!b44w9ibolh7pH?HNr1Ns(cLnp7KIxa36B>Y1ZNr6#i{hHjz= zloXLEL}i5Z^%fcuZ=4vPe@|I4KSCo!l*w0ah91GA-Sj8$b>(g{-KXzW&(hDCnqDm* zLX|(+1qF!A(sSmbYa#pA?(wN)ndFZ1W*+Ugh0l`5hG@44%FiER^)^$pPX&r6rhbL9 zmtGApP9{Sjw6<`{U z&}d~}-`Y_57PJ{1oBNSk%zr>Jh?6U>=yfRD7hEMK#6&c@O6bH!;&XZjFYXy-Ks2zx@{42nWj zfS&W-8QTX*$@iO1q(T#{S&_Bk)a;CpO`pOD{_Sj+m4(3x-hi?ww<=^N0taptCccsK zRP$iix^o6Un^{7*|aLsc-`ZC^+eh*CR5?lr0C9Z8o5|Mse)_Jj?%mG76II34?thz& zZgdt>`CY=lHLVXS*`j*bzKQTDuZFK#s1T8V4803JyzGwtMpx(fn$&^iXf_1MZj896 zPZz=fYOi`T(R^L{v2FGF8h^Sm#EQt>uj(X}xu!@HqA(l%K(5!&IwDpDUu7!Fq;}y~ zI-YBtTdhpU8tsjbqq1`G>c5~fw0gSymS)SYeq$f>l^%?F6<3x%`o-&`E*@^vKz^TJaE2MPAdSW*EJ0^j>r)J`b+9L0iOw-VbVd zv^!b4@`Du)@xC=o+X*~&WbI)B`fE)1bD|%mSFa|F&w@p&Db8%zTD`j~_d{Y+uj&eh$<7E@JbH0VW)fW91 z$&#}|S8Z!~^nxnYfCCF0sndmu7e~%AXnh)NpYM`#vV0};@0vy&ym?OR6t0h&e~{wg zy|!y&l5+MC2uQGMmz0dX;}&Awb5O;-Ur+rl{XREWc~XWnyWO?>KrC*6qV;00Yu-L1 z&WyjU1l`}AC$3VI@>O{7@R$A2mYeTcNAW#4gZMsdmzB4-qPK#JB8Kq`O;j0on$WRVq!Cd>3)t3{h7K=jscctDVViZ1;x!gi{i%3ECYFOKdm_FiVA&gO5jTSx| zOB`$Efn{}3IwHP(q8z)0`$E+KhZfckcZc#1t2gf6OG;eKsS*O?6`6jv%dN$$VWe}d zH%ZT_u2{)ly)))k8uwMGyY#x>=yEcP_Ex|(W}mR$9Wn_(3>tW&)U@u;xujB2S!mM1 zFlEgPwIRyVnp>wNPp-X)bhF~rBG&->j3hHxjoBmuWd-h;4z)^f_x( zaGK_PgX^m2o|np##>S8jxA2xsZDjp*1~izr-butlWk-DSL?-PyZgy|+yj!uvbALgX72Fe#!de3%^R=1=@gIi6c=XLTbVJ@5sf^8m(cy}k1G`4|s2`EJ{6UY~ zSmSaHMD5%S%hA8PaFHR(u`?a0jBkBB7>XA#yzjR!vXN5NejifxUO5y0keBCtug&kZ zoYE(s!lX&P%Dm*XidlhF5$X;n9Do(Ql#lPy0$Dg$s~SFFn*2IK=~b1L&FeUQ{ME27 zs(nJSp~d*3XfL&E$S=saf)9VlQbELD-Tlgn`0GG%?8-(6F^*wlVB`yj$(OSGNMyw> zeg2J@pO~u@oO<@~Ba76##siw-yGhSkf~Hm{N&s z4<+!@q)Fv7CB^!f@75llf)#74i)RHvh(DYl_#XELVPn1s>jEiaWtUGBrL!k&K} z)UB;#C8|e3x;083IDWe}bVv}WNRx^6x_7TB;?foMv+vKB5#&Pq$s@n#tA98k-aq{Q zbiab1KmfE_Dt-umHGT4BGp(Q0D7M~qQ>{6|csNMUvUyw+zeS%!n$D((W~Iur8;5K6 zwEgU>)ODbLV5kIcNu$uL1sb7rZmo77?M$mPbGwU^xfgl1^)t`h;;+*Zu3GAMdfBUT zct3GU9?L_v~{ zNU}_Ao07jN!;>&EFbFaKtpAnxUxEHj=8f%3P?bZ9;jM=fAZ1F38EW&Aj!u^8s`l3- bDj9+}Co@&exntg+KV}p_DzatL@BRM^K`Oyg literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/pinterest/pinterest_logo.png b/content/ko/case-studies/pinterest/pinterest_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..0f744e7828cc2c0643135267fa2a7f1d5fb0813a GIT binary patch literal 9003 zcmd6NWmH^E)-@751cJLm@CF)pcXtThG#acMx5g4AL4ySdE(rv8cL@Y{cL^>*8+mzV z=9%~VG4uU??^<&oyd)K;kBQ@0(urbLn5fBitl@w*QpY7~(gP|im-!C*f zET0XjyPToBj*E@Emn8&@AZ6`h1*TDQva|(jgDtIn+?R5>^4b@bIK`u_5 zmVfzhdONv3qY)59CA?iNL5^T|8Y{4^y|WnINpm|LjlH!PodKU3mzt{#*v?+j7XsGt zRo4aiI)a3(=_JHyM7@Qd37o+0mNec@@0_8+-ePqB;1z!E|1}Muqxpx6yQ3K0KSdd; zY0}8JK)^J7octUhZeA`L0U=IqelA`;#jTz@U; zZ$~w?|G%q~)4#o;?%Lr0)%zcbp}Ibf!+bJ#U;1{a>M6g=HXMOLrHDu8YgN zzq6=m=i=@HwR3T$k&*c;Yl1Y)YL*~-=f5Ud{((|c6IODDx?4Jfz)G@Wbk97T_V(7o z0&@JaQgZwPf&zlF+}v`4(t>ilg3>@a9%&(dSw1P?-&k1}kcSi4+5K;<_5Wf8|10*d zf^c$ujw}m?*n5JlF6~!o|ZQ$Hy%o^~?wS&szS8wf^r~`2VXGz_S>@ zU)%ftZ1>+&&o%T{_us|)JotC{gPosi9`an=2gu`D2nf_)lw_rJy%!EmF)Z-)?y>}I znUp?L{mCaTs2M{-NRD zx6vh~6%n>z$2l)@+_ub%mqrpj@`S7oxOTl%hueC80Sqlu+g?iIg%@x%U^?}xQLFm6dRrt_awh12xS(Ew3DL@f2cDU*SMW5{K z&F|+&#NFPPw|V0UcfqpD&U1?H_4szAZjm!#s)|ZPjW~$G^Gz?FX1GJtM1p4&d+X{bz57v zO=~yf+n9t^a_iSrgEaW?zSr2yxuq4Nz|@8#KgC^4f}4+1M>L6KHu|E)p*Q=ZfgpDG zl*b0QF2Ufc-(L&ejEss>QYg%MeMFT-HD|}#Hdbpo$dN|YXd#2e485k%-xa-2qKP2` z=q#LG6_prHd&s)AGMr%}O;FzvVC=+CH?1}T&6jU#0xxSXF1Gv8*+1pynbG!S1rfcz z&i+bPQ7WQipFbK9Cs0iYavI%7kOIL-dUr%2^WcoY zq0dGmcHSHRti*bLDEfl_n{6iwV9s5^Y?7xzeX>qjtLSEMb87yLqC}J(U4_;1!O0(8 z>%E#uUXnk4$bqqK9*TkG8vB}|klC;4a-+p36fVOGS(7g5UAxyh0eFde!>gTyuawgv z9Py4%t=y1`#@<#>jw%f@6g$?$TzRu_B^8p#aS6N36CU-ds?AKJ%%s)9S8s+cm_8Fy zqWuY-hA13$k5c->m#3Of4!w432Cjh4t@E{>I`ICi9E*Z#gl|$Ym(x=X%7cgW3KXA1 znfYgRqtzmY{7+xg7{`OV;mjoNf@u<_Oa{jw6#S{^H*XBdSNs7Nbu7}_R%bYDR}Vfh z2mKKEDXowlli{y7sTYy_6YY8(5m{?nl@&+LwNZJQuQ`JDixUPC1RQseO|otVP7B;8 z^n%qE?X;|a4S3O~VDpL1b*Nf16GYgJTzwF7QJjvE0*F!iL?Oa_-(4(tBu-BIe7w1; zS&;8{gi+tEr1u{Zv7^7fu=?fWHQE7WM4AW%4wbc+f^M%r8TIJBlZ3MlB40)qe&U@W z0>5;pW~Nq7@5n2=(Te1^tiqjK4@|*}3o3(}6dZ*Niwa*Is3rK1YvPv` zHy0{|486u00yuCZL=N1`G?4C=%$kNTA8@oi)Q%5}NT4YgUM|ZkkITZdAMnaIB^7R| ztT&yyKlH><5=yvH5Iaiv+pQiu+&V|w2H5w{P;#X6)0g8x2Q3_(CEmKlAAQoQ>kRGc zn!laLyk792u$@>YH%sV}-d}6RYUp{vJEN4MdPfT6>nz-P%E!Tzdz=4JixJir0EA|K zC0x;IcO5@|YDSLKSJs(3{BdaH)%!JoDJs+0pu9BT3JjYTZxu(RO%vG*N0pL?oZY#^ z@$C=N8chFSBeCyd{8FZa@^C!KT9;ZiiR0%aq2Atq`G`kTA^1S4n;Q7LC@GNKigrpZ zU)_c`@Zduhqn>Wg9_CTp!z1m--AT8~^igq@Y%AVYSN!ayP;*}n&z_1)+h|+n*L(X$ zxl@tH!cW0X9K>ctGgD(fB5e$qT-fGaFy!#3dUyTWIx8rF-*7u9I^7_)4qG z7%P7}LGAmlLkEWxZan(~(0+l~Xhn>`l)N1W>##A=<751$2|4uHC2ZkdOk5H04;3lP z8n3yLk+dAn(>&goc6^topf3(j>APMrpioab&l=Yp?(HWnw!dDE^;YI*j*)kdx0AtF zmr*v-CuHpagLfXbzsN*FT+ep(sx#v$;iOS06t*P%yk2~IQsgN~RuTt?QUHnYrHGxB zwrI6=`*q2oUP;JvzRNsR46q)$A-F06xBW7g^bjG8#X`sGfyPk%fx@|?O8j*6p z57k;6pT$gr8O~Oxhs0wI((EMw1a+_7$7(EN&v0}+3%|Rt${5A}vXEl~MOlv9W)>Ju z#q*C4gd#b}SERnf*alLs@Vbkdt`0=-f7qsp&cKz_P9-`rmb*+Ia+dZ?tX?EhmbCPC z2OxH!BfM_X5fIsF(-A-NEZ#wQ(^tua-{Yf?f}gf-H%=Gf)&_gBy_hF%RIYzdzP@2a zx-b~U?TuWiE9rc*H(7Wso}EE8V_nG7S@HET?SZjK?Vb5KErMNq zckN8yPV4pX0=(xU-n8y%y1v9l43+$?rlv-^w>zVD$47oXl58#W2%;(D0|}yJ^(L}g zF9vdwLD|WWHqAs#Y{n)fRG)z0oZBn``Q!t#)V-z``~x*CHgj>Dh5EXvl+$}OLTei7 zx6P>)q_g9@E!8wvzqm}C<(onZ8e^&`$J0XB?+T?kSaEF;Pv&b1&{DpyYty_~Um|2e zq@wk&=DQ$%;ea!Cr#k(nV$hIdEj1((wdO~Se?^2XsakMUyG%SokP44##BK}jTwV2H z@pooc<%TO`cq-jDk3lK->X$>ie>SV2zd}*7DLqbLCK9%s?!L@vy_IRJ8ehEmVFL_z)7D(orwG+_Yh|<(v*zfH;pJz_sPs|XK1(-%?y?fX^Q44W&zZn&;!E( zPc?gsfa<5aPnxOWz*maK5k#vWq8H24KQsY<@BHN6?AsFyf`Ru+ZX5w8z@xi-mf6cX zqE!)7=K9*aK%7xX&+9G8L)%aC$fV+h9>el2&V&}gFWi?Oj+L0SUc9uO`qJ%VACWlu z*yOBm?&gUTrsdVm)hY8z;S~fa7Pw zmwQd3uFFc6f(wLf(-E}VQxw%SJ7xxAP`T6ld|G&k+FCd%j}!2qv2$;d-IaVY`z|Yx z+Rk2jgIo@E|FN(2TEbKi79(z|htM4$?VyxoMv0U#<}k#rj6~tfvW-xvEA}J`K zN7EVc| zDK`t(FVA`L`m2e@l#s+D`$guePf4+T6i`^XfR~B0*wtciPvmsCanEF~E5|ueIymfD zLw7U5oywuV^<}vidsqZp{1+}zyna0)dS_53N3f;^jvDq21st(XA3>fEiL?;j$@R@s z!xdq$8Z?{3M)gX8G-&Iit`Q+8msQp$J1abvyjk39y=aY-x2FpiIX`;1hYF=w)wdXN#qr7`;*LR3oHT_UUm5u>+;lL5pbZZwMIST{A@J;6oV;sBSM_?i5 zxW?iip@77ItS6W99h_NK-q6h=bDI@DQ&lyoSz?G(NFx<7I$;(?(4jMbXyl|0 z=(%JVt&Q`!SX}$88bVEYs>SrtvO0b->01B9&xbSDr6=`tY%^4Kpq88B#JDx4$qSXI zlBR0~jnI4qbS;;Rkv8jOZEx1yhC*s?Y$0`7;k2{}0ET zT{Nz5-*n@0{xtR=Jk_1I3arMo*$j?W|2ZM{yZhF_O|SL<&%wQD&-LtLkR|qcjX}Bq z@Ahr1jaEUw-M*PrI4L4v0JCJh2<5@er7PS4vuVAIEy^Xbl+ha~V)7WeVj1QT^dk-s zAF(<5e*QiO#dO)zp}~TEzAu*ivi6x>kC5Oj;=ARReJM1?L_E43K(_Q%7wIHfi*d zmnDv)?wi_of{Jn}unf_KtKZy3v8fTE%5(Tn+&{)7ls2s(y`E9T=b7-+Rp{djb#R_7{*2*PJW-3n}&rZzYoesZ249nY*swZ$e^8+v0~Ez zM$CGTT=lNzC0JC&)UWuL(U(vO|L}bf){W^HakiH1%gX|#0d!XiOlG0+!H-vKO~Z?v zt%a7Ix#n|3ke^EN^p=0)ghDpPnx97C6ywe;%3XxSICuReroOkhPOjSHO4Q<<{Gu1p za*{J=o4oh(LG&L@)sm2-0#xH44#(T7IM$_csfqQTBy-;W8tdwqIr`cjo;IC(DvDj3 z_ltEHkb4_B67Sh53okr5AYw0u+kJO&;XC!SR~{cSWmmr=nHM66p>Q{lxRqWa1!)eP zf2wJ*cQXny*daY=NeS`nT#9lcT z;>lM&Dg)&*iNcO1)x{eXZ&g6?oVsM%L~ZES_bWQ_9LxcR9c?*OYF~YJ0tl| znqG&&>%_i$j!09&yJd?{f{<;zZUsrT0=;2v(U2Ik?E=BKB$8e;H*VpQYI#7ND(X~H z#`Oc7yShZAB@t5>s3i6KN1vUHL#jDSP-M>4P}Rto>)3#~^fYtC*+Vt0wn;H6Nyd&` zaD9=#hV-amgtR#BPy13hc3+J6w9tt;1?QPm2POx(ihb}p4h9Ra9tw?UWjs~a`^OQb zsP42mOBpMXA>860{G8^M@E9~p(C5s+Et% zomDOnhi+iZ^A8=Lk!>a3bN+l=X`sibGCt-^6rpmAoP=OeN*4VYZ1^?54=44)mRp4W zykb9Prx-0`9lic@SMiucHF;@l0S?S4ED?aYcJFjlx>#3~;Zf)@U~#b~^_ak8nruhG zb_`pezNyS>0xwWpJWXrWQDX44ssb=jZX1ElIkzC|%|noan-R;{Z>e*FNSDUqs{5ZQ z>OR|z1E6BFCwN<r}@<5KdEKIr3q}E~x7~NMdeckb2g4#jG5n)?%F?{j&mIeZtF@MrQ73doVqz ziN0~~%F+}b2JQ2LJC96Or*?&8PnF#XTW5t24|AqI*-YeP#P`<(Udt>Jn8*}%Q6=V$`miXxk7BF1)#!l=py_#ZB3|{~WxUD3wz9dYOm>bylyh`3QqK zZBtXdM_Z(KDD>rxzK4(LBmlm;AMQ1%R9uzOcbuhZcncm3T`6g@kQ%VYCm=Qa%C2o{ zWovi1x9bV&KNYjb zto!?oS5U2JK0;&PnN;G<)Jwm5{~-wcW#<-aA=$t8P0x3;tfe`IQzY=tYb86mya~K& zYB+tB)AMIw_x>#wT@c#7L-NM-g9qa#K&mo^QHW)12a<9kWjv;+xaW>je*w>Lg9}Bd{_(L1|Aqlr1YhV_E)%h0*r*Pxk z@5wI>8@XF0fXc0HD&5YTI^bSamg1VV$j?#D0s zyA?BbI4E*a2|!K^ctNSJ6CY>fqm-NPCjP)5B^ixaJo zh~PGgnTKD7K(TgPT$zVs-L-yj*z^dfNEy@pQA!2AT$9B*O0ia;oSqn=BOYC&V2?v` zFg~ncJi$^j*h)9z%`!{czyIi zcz0tHZzP=tE=JSh5Sc^T*%dLf;`w-1F~$!O%t|jG?4(yy|MqR8;#)Y9*4%rWTdjq6 z52H;S_?!lwdOR5z`$N1<4sL05G3#wILKk;6N?UE25J>_^{U^{xiL(qt&d6#0}LywmtvWp(!hcEY{W*on>qW)uSN zV3AKiA2HA+UHkO-t^fE~P`l{`Jg0IjtE&&j!P5M zcfXq77D}ilYdk&(TXfD&I4stfK$2?dQ8c&11{=xkDp$>yw`JcngSkVox>g4R?t>pk z#iXuPQtmm`Ff|+6Z1G+I`_nbYg0B=4(5}>uu7Bwmd8y;T*iLTyH_Dn`F!koX%^t(P zLpJV+mAF194d3*qnm4F*8OmqNscyXhzog+I{n*6fcYOHktqUf#`Pp7p1~p&+cAM05 z(Wj8c%Ns>0;%Oj$YXvtjhlXHF{FHvrhA!Vo%B3{ZB4DPJ%JsOHGIi4nsjcfI0g}Wu z{;KKuIC`>v^ddctoS4&Qm`E&Ko>sg-tu?+f^q7}`@~ z5aFkMJk4#%)y2ZF*`(}dN2A-XemdWGl1?XIIdEA{G{*B85wHuuYI85)mr@F)OG~*d zthUvVfX}5em^v;EtR^u$W9OIc1rBIV)~Z|CTwH+lh;36bfPUrd?YQ<++e2p)s=@EN z)b&s`B_$J2`SWe!xtPz(tSY))z+UX!? zy0Q>2mBsVv&Q~B*Y2lffryM`uBr=glFQD$z?R25n$}HKUZnk`xy5yUb`{QvD0b>YM zm@lu+?T`Vi@E-3^Iu_$^_q*<>pM$Z06x4N`n0;7glY4Ar44+DzAEi&)KHv5B_4{rE z4)D>I+HBbSek2e-&Rao1;J!6Kw?0fP$BZD#aJZmhSxu$g+B;`=L-EYRa73upZoa?$ zC$S^${U7$8osXdKmP_S6`s?wZ81iw{U(B8E{2NU+_v~lXlBKlYEzn;_8(z^?hJ?Lu zm0xA~tg&3d<9xoYT4-wT@Ct=BxZ`nl*R90}HZA-jCs+Pk-}tRI$HS2^qR^TPabGmm z*|6i024SrXobvdZGW@C1 zSc73oJgf%pQ#ms$SA6d^KZ`5Kl>+;p}sc{@3p-N^Y;b< zT~!gviqgmk_y}NNV92sE5~}~%mH&o69L#^S54#rEe+JxHRay+Jc82imzXg<)sDdaM zSVJ7*hY2(o7@%8LLR8(u@X7$zSY2aXa6xWD-s?tian4&MGlm?29Ig|GQ5$VIGNh1H zQv_X7MM6|m8LHy1w>??>wl}*h^}@#Y7Jr1)aLN|vH~w>uE7xGR+m7!}$BEC&mV4fq zX+t}yNI*az;eKf!1QgtB6j{7nK!7Gtgc%${7apPm3eK7e?hu1CKZW%FaijJ(7);R* zZ$+&BYb?w0cYaY)ErQX&gN$~^bEnOw)1SRtMw^dS0ez5APAw*cd}(nTdG9b!=XkHy zoat@H5EC}j3UbgNZDjFPF@<7Q{lUBh=x7`8sv)of#a=uH>{B0yNZCNa4?U6p&Xvdk zMWVC#!lt-Gkv>R!UpWdae4#^mE*{(-NmcWct`7d+Eo|x4C`eg-L{Bi-SQ8?YuZPCl zySVd;dJ+7lJ7HmBp=(AK`T8`ggWp{LxkSd=^fGteG{(>Z(mH~*&p<&8+l<1)V69JXM!{OQ-ryB4?0X!vy|JSsgp zc{lQ+b4r;cYOiK2brZ)%`23?M>(-DCzMB;EiNUGS}P^@C_6E z;@m>6U1#7~1CM)u(QqRVjgnU<5wsQ|D`@_D`4Ced`V>4ei(0p}@*{{3mh;b(_vQ48 z+3JS${QO?PbVYx$;?CYb6FC(A5E)PrPrAA(Iu^REkk62{qz+9u=K3@5@4<&dPbDwBMAw?x_a zB!eVCNjXmgHam;(Yw2U~uxZ%zb$?f!+u$w(+NF#m1~j+ipH$F{bQ`uM>By!Nzcu+gApg>vOkUGC3?8 zGa*Z;$Tnp91G=osEa0zx-h_!b{q8F<<7@i{TYt3$bcCuo0g_QNI$D zt7gHIq9G2VrA{dT3xP635EM95v$`w5$1M;#VC^Uu@f3dA z;r1=6F}t24TOB^Tl0B5?q>RE*AI0N0LM^;giY9)&g+IC*5rdd$Mj(U+0>62D0$RmU z%&q34cvW>^S~oF!L<(%#+BJO68tRiK$)cVQ?+;2=vY#RmG}o5_q6Xr4s9$u|T>O6P z?!*@;XYnewq4Q$T5xTw^@<=t^I0=Spr)Ulc20^{zA!OM0Ietm8CZRP22pE5$%*iP% zU2;=$A1Ng)Bm6T+tQZU7}R>Jp+j3hjDu+_m$=3EG&5`x3$U3|*q22!NS zFhQcJY`lLF2mgvW80eA$o9*;oFB5A#Zn6f$<<8+k!blbqVW8;bcC8iL&IqZBH8qZv ze`<5q^BkqG9LVaEIA!*#6-k^l7Z{^$PkHZ6+DeWYDOtB)CQ{SB@bjlz|9B)BkIFJ?3=XfkDFyXb zdU!<}wYZ{02L?W}<>td7`z3#B#H|}7>C0Tdo@Nwf-oq)L3t~XPJ}xb z74(}%xHR7FcGtGKg6;R5Hy%zSJhVpt+ecRenLT;78|26{>R?o{DSHLnSsH@l#evZP z1N#qCXgb0peU!p-b#saqw)mQsL7TL}-N)c*kjr0r8#0hw5?`TzC90k8K4$h*rG&DN zl{VEE9(DtYS`zM{yz3m&K91(`R#*$PT7JsX!3G;QMJlhluC%nph7TP|U8#RKoIl;N zz#5S*YKHgVj`5up|Lug{&H#%L^m(ye{YK85-gs%GK|JRC;v?s#bnK9_(j!Gl3`H9n z;QRS0-r3u)ra&RFs&FLSzV*oTs+F;kY!M+cWv<+2p6(UuZPZq>Y>u(hx>$G7Lrwf! zE|0d$@6t9)2!}j zFGIErR2EX!QZ(oncse@|)XpVce=j42JQp;jA|gf$?%endt!WQ?zJ__i+gf;eZ+SB% zX-_S;;_{YNQ>0j#RDvQ5u>HKG!2Wx= zS}U{n`wHZgm4}kHJNy#6d)q;8Qz5upw!BxGjORXd+!F3_Vg`k?dcaa zmiVM)Fc1J3QmX_fvDeMh)lgCqDghN=32iY@?n6U2D?Y-bqTUsrz%h#K; z9HE&_T{t$iSzp1!&kKy*(~yC4BHu&}e`F(D zZ-Tn&iOo;C(M``W!J3mDtC<_5OB}S|6g?w=z@^fCas5Iv!8c{ELe%6|N0rxMAYfdT zJi0)O57|&_Z8Cj0VpeVG^x8Fkf?J&L*jl9m1FbWh6w`1_Hhj9EC*zZ}3y94Rzx+Hd z=?P)p7A|OW6ZEofD-5E}x$iK2rAHdFr^2V*nPRFN1>JC&U58GtrbF8D>jla;z7*fZ z%MF=(4<{Qn4zw~g|8Qa}N$C(l@3AhL$vhD|8C3Ab^^i_K*KrHjs+qxx?po+IIOH+`}*Vl;9hC{3K` z^XR39Kb=VRi`$LP9YS6^Da-Rj@C>)~_x@!wHxj%Xxz(aYt$39bv?hwdO^9EPQ#BW1 zY~43CK+WySAhMCSZNq=4)eHD(!_#doNatuEe>W|}PTv;!)p#X%%D<5OYzP}(tf^nA z6r^5Js4s_Drmg5|In+ZW92{CRG%?aYA-(haoS>o22gxh6%*^3(FOL9*`b~aKY57^y z&kWCt#^ck({p0SE-j?48A*C!7pXn{0UJ@!T@smOFr_Ld54E%T|gv)Zan15c5xhSHe z3^&mI*gTgHR>K@3n;jc)KFp(H4JzeW()Mk~;%8PbC?5gx73JKEvuSba5Y#73ho0HT zj7VXQ$Y%y^y(@~!VwgFXv(`@XP4!+mH%FCVYj3K7ealz+!(z#^tJbwUD5HQy#~4Ai zOaafm2vFvSKba0Gz1RD}cm%pl%8+%GC=J{_7-Cr0>DF}|8ZVTNO!}G({g&t1*{)!b z00PqAyIqb&6=ZOz6xdWt)78I6{khhYJ@O}Qx&g;?W zE|iRBYnb$oW!Xq@;U{8F>rQ@6Oji4thiTVR?gP;B|EdT9F1RqG0#hd|R7@3TjZ^}y z8$*Ppp*K>`Q;z`Pkr=DvGbtD4PyZmtk`Y$b&EK(XXt9`8ck$#`yP?A9VZ;?W%1|q8 zWoT1kYIPh)6BInL@H1jDjj+*C(%E2A-q4(L8S3L+hw8}9d#9SwXD%YGEC%BglY+koeSlb-EJbe}k4xI%_{K&)?P^Y7$U=fY^%UvgtI<3rK#&6sMw$d8h^#!D%pVoWwY8Ro^3 zU=Q1z>%CU5Kjd$ERu#H;QZpv{o%-99`hpLzKWXD7Pl>(27v6pDA3CuQkBPT`61@0x zqcSKVf!fZ*S)Dhc((ku07V$SP;`t>xp)dBM>OcUClb6cy1h+IO^lgiHOcg&@+qdg$ zjS}ORsgmp{ADtu{N9EvhYaP;rsEXl?kItQ?kZoIO(G+-1SULo{l6FeOROc|#-{oW} zKgAj9+_?R0`xKrEsZ*YcI#H6-lAI#RP$B~)F9Ug0TfGyxjlK5CyEAWDkEx=E7~tbf zoes5>RVn(tSi6SftEKO)<{Wj51-Olc1nA2BN(hmfW0D}Am-2E4P=5r`&TWYo zUSBj~U@U{pvGa3M)?N8QY&gT1j7*bP`>02O;IVxUCCaU+XuztXs@W^+{^Y4K7m@wt zF=ieM7%VSaNFrrdpYr)iR>`A;d;^`j5l;Mn6mEuSN!TVcW-@r|K*<}dEJ|c+{AH-)&{2=%lq$ApAxC6Z#=SU zBPpxRw4ATfVyUn?Qgs4s`BK8Wg!H1;vPcH^4czxSZgN36d9_2eh`;*KqHTt=#*Uck zfZCK&t~n8~Y8=&EG%j9<#Fx^oR@%5Or;EhQ5g5!_A*S=Pt;!KnBg53grW4rGZnVL+ z==LDJ!ZU=a?SD|c9sT#5SiRI zjkcepAHDF!CA{*nh=QlSYY=jlQtAQ8P2NcI3(M{Ur0W>N3TB`US}|i-=!E?XLI`gO zt}UoQxjhI+_jXt%RB#=`sey^ zyU;MncBTIw@Eg~$r1DG8bbSc?7WFwOqA-6*Ra(@(3`#2s^;!7JFi?kod4gN7dj|+H zCK*^GLcKe?QR8w4kCXe$jCC%K#dE+WLW9(h)SO2VPS~sIh3{ z-#`T$*X;?#E$oWcHp3S~U`agKW`zxRx7=3d6!*LlF3$o)OxhX=6X;8?l7JduYmF+JftE1Oua+R^iH`ir}{Wko(+(ynlD-Uf(_%<573v!Fb7o?wmNfU$Oe*bgw0iAz}MCqOR%Y zt%KPQCWpJkI$XU(T-L&@NEAZdRpyEJjM{y{v*{*d(K>s@J#f)txZEzVZ)0f(`%kDo z@uA+f|FP!%Q?X&Pvj)HXt!jGxr>PO)bYxJV?ko2D$nL1Kub)bOYCa-WAq)$1Ky^5E z7IQaju9pYKb%$gW?T&5o#iv7Sd7B>eFN5VE&BC~XkAVoV8U&-8vEe_nH-kWyhxlBB z2<0V7m)o_F89!mNwHpBeff9$xqF-j{@@~*6NxiMrTRTvUacF$Yg^+5ud-He7%HZQ1G7d5oBck(2ToTQ@|pIAIgK0;_!7hb6+oWvk?_1 zG1-K?qCTchuZQX~+A{i@K8)L!i$R>mvseWu)E7O?U-FFRR^dsyS~ z(i0X%mfG}greCpTw{f#OgBKelLsrr>{S#iZwJw&AdSG-iHoL!0&#Q~k3{MyAv2@rd z`e2LO=0ePHlZCLoNTpX|b)8_DaToV$mi`s{Amgprac&dy5zE=72l56PjW+chA5<{U z?swE4IH*l&6{{+)$I)p<7@-vc_Pq1=q z4wujsULFNv^~rbNO2VV$O%A`e351lR-DY(w+4>{$BYKHd*$(y4$=#}sO0$xB@YB^r zo&zSl5yTfYl9lFr{p4~jS^pYDctF8pTgV-mAL8woS948W2;w@2`2wS!4A|U0sR`*6 z#p~*Q;hxC|r1M_2$&oot&49h2t39KB`617nOZ8RVa$Xu-^i+2n+KDm1e;l8=hLnlF z+X_P!E;P=CvNf&;SS|pQ>x%BM=q~}IUwp(B+=8m&Ul$WpoM*`lvTVOLmDn;V-BnM< zUJLT$?M8$M^yDsVrJ?Ia6F7v@&xD+FJe19hb_i70bY!NA(f2SK6MHG?HjPiFF)!i? zgyFE!5zLB`cvbd3H+wRnPd41Ol9=QJ9u;*G2=6GNWC#D<=T&#~GNvKmo_)TvJk=DI z$LzHSG)?5bgY%G-&rctO(x(k@Bl7tkM|)%SS9}CpDjjQ@967LOELBh4bxWL6fAgX& z(FPx$SV}?_|RhwX(R}+HLqs|aNsmGQIkr8vjV&V*cfVyk& zw-cqO#eswde|9bDGABvgvPGLvx94_>WLCsHRlt2sYX1GvVj`8TJcXP!sJ!;iC|StX+_TEsO9Se zq_R;!)!1lvBtxzxgLcFuTIQ&?@ANe`l1PEt54gV8I?BhlvY}v0-A}P=+n*^$ML)=i zs>VRv*47Kx;72S-<&W3G4VB{@!dO;T%UC4tzg=Ya;lE}^Qs6Id*WdQoM zVFae-d1GlTf+0r}VXP132WRa5&R!G+@K%{H}lR~Z>O5@2t0Vu#39tp%^sBCH36zCg2 z4GH)zM2^tzN$(Oo&FuCZMiUHuMRe7m#@n}P=T}1Pf1dr>Bza;QMX1E*PjuvT;k>kc zd#(-fi1*O2);6s7?uMR9g9}D^q z>1P+6?lcOo>Zdc>i8tsI^D3!g_9D#aI4Ip8#p^ju=)(}_Q_uh^yYdR$zH#Qo{UiOH z(7N0T^GJp)p*=rYcYIxO0M?r(abVc=C?o6EVp-g0it1EgjVZ=h5?!MUvqZ{NZ^181 znDGWd!<=l1262z+foXlyoOG$0Sc112E=cubJ(=!mm#5b48x$ZzL!skIq5VOKYtlhvrjW zN9;nBRk0>OS@*NDL}Fh08~ZC~p_ohKScLN5;k-~|2L&n3e{7*hSq@npi>;y=>xQU< z0JYi($orlbJY7;sXba0y7CA9<|I>|QV;n<)-W_tOq~KRR>;>Cnl7)%Gm|^b59p4`@ zH)NfGQ0Dk}D#9(M@H|~Tu`Ej5!JJ=#GXAclse-Gxh^3F@-=-zOYZ98nGMa;tJvv*; z%AyQktqny{G)0K`S4mL>s;#DL#j<#ddxWT!NR2vmwt3LFl8hIBE(o0Q4T|Uj^F{s? zYcASgCeXL%CuTIPi@$>puBqmHJg^)d50I`=gt(CyrknW2JMESON{>NogGQ8n@EN#GKi^Tjtz=Ba#zPm z9j)QvMh}6hRO3)y#Dk2I3&q;8&828CV{eJLs~lck#g?F?<0Pk_2s6-kTnkE5>+%a! z+#5pTGveBAUB{pD{_&Yi>G``DovkU%!l)HkN*qy_%O2}0`vorEz0u* zYOtYNMxfJ#>U;{HcIYw?z$QicdAOb1Y9dNdA_z_5iVU5c?Y1|NhHD2Z5}p|aVxofN zYo4A87IiVTe=`*<>xv~y4Ugq{w8*v)mFWc6POqP3*_O^rwaUo*JKn7ud^2E5un zEJM527?YmwYdnD$xcwBcVO=jsfTltelA5DVY^14~O{=&CCGfM_4XO&Xd}1Fij*5tD z#tLquA8SO&;ABk`Q8pvDNYZjkiBY@m$^3#;zHCGODX>7%vz!!hLWk*CG6^+C^ z6*V^*E&1=mOi&qS=UJLpX^xNAn0gx~EK z3)QRq)Gj~c)W+W{s5hG&4)nqJ5A@43MFXoXp2L9)l1yjk6x_QFPxeJ4yW*ARCB_&a zy|wV<{-EpBTxcO^0q^_6=VS{BA&!<;;_jAIWrxOa@8Yc!JT$-UEnT&mWgUoD4FZ?UrQvab+ZU`DkrGb0;WsoPN;xvNJE_7E8F-_ptF=D? z<;up&hd%_iEQF4qJYjBjSP2d~iJj}_ny<$zdC0ZAB|K9LtmZ_2CQ?mtP-FaBYnO*W zAJ0PX8}41R?{L|g;dBX0r*w6BX|;m?@n2zvt(Xdmj8h#kY&g1e#AYI`8mm2U)#uLA z9r0UlZn#;w>TTj!n$>;dz-KHW8=_)FTMkDGsbv!i8S#x;!$jN>SmAnw;{>(PB^X+O zzC)qmjqm&EZY&y?k-lVhHVbY_1w2}!c?bxZo1i%uX$7fc^+A*4<{UO^ z94V;)hgy}TWGXP`BrK@+>rMU)qH#|YTGFWx?uBW1`UqCqmKVW~nOfj}E9x;e zeM_EL$b}htNik=%8YU#PTQ^(A#rkTo1dD`l4SQXq80RV5z5s{f%-YNgJeSX#Vp3heDAbgDp3V=%4fzu*=X4eW;rZ8Q!d6 zBW}3mf~r***=yxTuBl z^6uR{y!OKkrBD4Xv5TCXh-6G|_z^t1r>NAe(u{n6va9``#l)6oE@2ELFT6}$Oz;f4 zOoBda**I9mVxW&1#1Rj%&!cz9JT4BO2wqA-+zzvLHUgu75VFQJS1Xk+7jbp3byo?^ z=s{2vs>8j|&L1}N)qD%n-z6#%6WH>j1(CLP?Ye~(jyA2q+@;WepML2skM!5hazdh} z5Pst=1)UgGnHx>?!)^-lcWr<8TvjEka|uzyaj8MNbeznI zP&R3t^vg+6ngQH=Ag)QfUYD=ERm3Qh5)~e)mdsJ{Vdjjgf&@QKgVV|o7}DeiDNv&& zbVqp_V@$Jq;-%eICZq>A>Oblx_TvVR4DKY~YCQh!Cbj!ds&+fQ`KD)M?S|rO0~+Z9 zB(IsB0UxX3C_R!fUfy6nG9PwuUuYGXNZEA}qn{gTWIf4vL)BdGtn3Y_6ojAC)!@Wi z|L6^cd1H3E{dzapUz))e`|AT2Btw~5dAi$?zlpRBIOMdK(1%kU8fO(BZ7Jf?^x;>q zI`B%Puc4t0C}BN8QU>tJP6UeBW&18J?SI;n+KI~hcU+b^W4fNA)R7AhsoDJogsrM-mJGPF86^G; ziO6(^IO|C+`L{PXf-j1Itwnr$^oP$s!qDQy)^iM=N51f41l7On7PT6f+?m3?>ndnEXRN5wjN+__Ih$~9U9K+CG4rd zvej&`2!>T-mJDG*X334ffAG(_QY-|DPhqT6of6cAk&OPYF?p{ZtVy9@(P7rg2O*Msorki66b=gt5}@(}P2Qs#q=Fou+th5qcT$ zV3IUo>?i)Mc8Aa718mm*HIe`AXi`iTqsU4N+*0msbUI3+640AvEPyWRguYYEZ;j;Q zd|y~1|0xs2(zgbguz*nU7cv*$2x}R}*kO}TXhE$hvtD9Tp2@rf6w37C?wr!Ta%!x$h(Aui_>}{?^M`1<94X1+K)EbX)G>(YyyQQ8(VPzqWbN zrszjxHDeAE#|+3%_WN*5q!U~@I76Qt_x@QU!&X7j<;sJp?cjoq|Hlw-X%g{brCQQ6 z1Z7*4ZEmC##C#8o!GFN(BOiJ{5ihbmg!Fp&Js4jU6|hztDq$NiRxA#w!#W#|74C!K zpf;fBf+8FvX7Dre0y04hGuqUb2g`|0Q)mZQWy<~&#ujHsoTjnia#E1TjT{pNHeqs- zATj`MY|UjEL#a_ig1phqkQl)>mmlnVoY3FaX@SHoS9He?KKBD1aB+Ujt=0v4ac!_>a-8nOUhKtNzqqZG)nTDgo5$lt3J?|7c35dAC-Aqn@FJ|# z!QLVZ8{ZvdFngFt3`PzLG4CDcJ<_~3rw@MUF!6UuQ&=tZ#Zpd#KmKyEVt))Bf{%+) z7*~z{3S|f0Gf8jwMz}Gt4e!T+w^%_8`BTN*nh zP2#X-Yvae9hp`@>y)#U>x?3z-PSNDa@^MqZ*xzsu*Z}$c{O(b(vI=7Nm~c=miTnP? zn*3|oa|j<#YB>3E>Y^wHU1I&H*{V64{D_ zub8AL%Zr&3F2i00hS@{**Ef1cCg}2L^*wF(w1Ux^+?1z7qF98y1A)0Adq#AEgxAC~ zQ{J4)DT}v*d*%{>gvDL;?Y211@zEHE)fA0TNLRatoOTH3wc;GDyf$=GC$6cJd2X%y z?%$j=+t24imcv4|F?z5s$5UthztX!0-f9kJVWkAqf3W&qBYq=o@r{x(PQ^LV!6Ipx zYmJqsaK4%?46|JeL-fUnK8kMJj&?tPY~8IE8yaJ3HD-07bfv^8x$s8E^HuHHjs+I6 z6O9b0-L@R=r|=)3fpUjmY2qh~PnX&7Z)UTiTS$as_g2bO6qm!_vLEKG2tx`OVX1>6 zWagVe{&36MkEevo-+&EsS8E)FU+f~3m!Ku3?x7F{`LGXkH_mknHVb; z&vw2(3hIGI!uwDV5nI508D9^4#+qcju8W7ct(D6Sp6Am08^_~CLU;qW%V2wgAlvit zwM-#SWrnJTIz%;9sm)&0)6i*A!oNBxuCNXyh}+xltDQ(G|?v= z;bFI$E~KOz45gC0BH~%JRfkfopZs~~2|a91h;$>zSq0Oc6D&_3KQ75NhwbMQfms*y z?(1Ct}ZQj}nF z6vOrg!yYzp|G=*ZbVSPio^u{UnlO==gP{?}TuXr14jmWd$`>OEp!oNz+?rBPx+4)_ zDOxlhedwT6o$gC?lliR$x(@;l_~>_gpr6d25sLTj8~_?ViAprs<3@&%B}2goO*oJkoUgBb%W|iF zC)%2tIAUUA9zk=<4%10dR39(Gt$2EFxdCBiFrpQZV=oLQ&G9zjl*(gQ&577J;(}Ai zobA2dw1yroco3g!$PHDnh`XP#&t~7fzO?krq6R(aPfwPMr}9{rnvJC>hB(h2(EzFv zW0n4u(V)vB{J=oJ(;a&-7!rTfB_aX z(>_v}Zr{CRZxCH-V#&|Fo``Y0wUAk*xnsxwqTAb@muvqsaZ!wC`YPcTO3fJ32N{Ko z&VEsaR_JK*pF2p`J1BykluoW4x~!8e_TPThdN0Tm6IIHp%K_a!rC5r=VIia~1_BuS zMd&yfGT=-{2&(G(vDKG8`kl$*sbx6eUjPZ|f8_B_Eh-e!W<$5=nYGZiMJH-$ph+m^Aq#$0P}bv%*MP^9CZ+c)MwE5wH(+{dskWNFb)*w@nrzw@8%+fb48mr{YrlG~vKCt**%5C;JF*pEO?`WPEv&AFQ;{%& zD%8r9)eFl$(-L|fMqVAc_n#XE!}URns(SHxM!@xTz6o#nZVF1${4Tr)jzW7**(Wbc$aj4u}E*OJ#$&b;F>mBHC%tO=Fw}oF*p3cZp zC^v!Nz8hqmbP)JlL5qD6fB&j7`dlEJpJ44zbvP|9>D=9lyYCmkUXzla)t`;_to9y& zp)(`mM8&{zK7=5Y0NYlHZLkwD%p3=iAz9rXS+~2MkNFwu9S<~%i(kpvH%VrNU&s@a zzFwhs(wL0~e%H-X+%J*}7U6Sm@nF@>`eLB(fnMzn6cyKtMrR7Mu~LqJ;BPCj#kQiw%3Ew_GYNQeb^LT|o&pb>Sv=m?jGZnhdq>`0BMaXN8lN2Xxnr4~Qq~po z2@fylhWxL2WVpXuGC1$do4&upjB~I!@_aLGjRjFim_*doDSNdW0(3n&Eqm^X-E24F zNE-v?v9(#dygT%hf78ik6gr;E!`SDn(IRCH4?y}35OE*;vns_fxxd?`Xp%7?EH3{_ zf7+L`PFtVx-Nk87PY7IdHWW1!WdZ}UGN4>(7 zPGpp{U~f_sbj$>=4Ew3eV#Szp#dtrZScxvj2k6{|mCo#yJ4Pzl%iU`Au}#OwSH>c6t4WdIR0 z7iMMob-SY$TX)rZxkq{$$v3PirvK3`i#cmffi|75yA?c-ElR&Zm)RK>GK_aJhO$To zH4_t)h50R7TH5@b6LOk9n5QRp+qGsw3j1CZclpd%cy~o<7`i0Q3uw-SRH8g&6cl&_ ziC;R{G&tVZQCr!TlON-SFwp-@R%$8m1YFp2z3={Xzg`BtdaPZHwd;og2W+M?xU=%w z0{oG`IS$_#`jby{+;^mikCek=56yP-x3&A+&@Fp9;V!srDef!xM9s(BlVdXRiBN5f zjF>uIFGQI7guR~5k?if2h*yiGQ1*(}GcqNm!$3)XFHZ}mo57eirI!Xnc+cqqYCRT-ai0tNHO z`E2kSR_&uNZ4;uyYbdMEMJ=rQX7k#hitN4d+G2?#k0k1y&ostnm=903x>>-r!hXBG zs{JP%c*fv!6ONkR7#&*>@ESB`y#qsugOl>| zqDF`Uc(PV40UD#RWn?484Gqy}$6Y$Tj+g8IOcHPU?|3{G30R*sx{_`uIN{B;$Z7es zyfF~+`4C#oYMD$|zQo6)g%nU+k8~iWMQGcQ>ylQaa<_#l&iU#|9b1Qjp*k3JBXq4t|mQBmbtFk_zpw+DL$`V%jxI{#ZPyes3P$>5O_3247ky@h{ z?^l8Gvcr!q9`*=2SU4dIS7(dABsy?5@P*V;Xk~#}Hi=J18-%y`nSuTT9WaVb#!r*S zfXl$~9n(5JLr02{wRcuTcC;CpG0QZ>p|ll6EZDiwh+qABiD`%mQHuicUl?6(H)0BZ z1euG|_8O#bfVf8-9C_ZD^a#4t+*3d{3uEJdi)aWR0bnGAyiPDr#rTXKShQWSkV zLth3;=Eg*bj3j0+tz356k#1KQY+tdz&^m19x~yQk-)FjH6l90rZkCI>3v-trhBviH z2Kl3e25SYw=(rhn&j+7Jy{M5T$fayyT5og>h;DM^!{hcRN}led%+2xoZm_WqQHfIG z5~?c!uu)4BebU3D@z@{9i*G#JI>j%BznEmeAc$9GN-lueO6=^JHF-4A3m7@MIUMQ(g+|1FPtr!4{a}Ncbb^+0*e_J2 zC@i+i3;_gU@7bM9OQY&9B^8kn@E`}UHDrzPe7mHrj!s5qhQ!65I8XW!M< zPcV`n=ot9nAV}}sH;`}rLz#H``#LOK$&Qf07vGVp6|3(^RkH$E-`W%-ks7fSr6uy3 z!b$Wba=U{UhmiI&cCPm`vd4VG8eglBcngN_ zoUyY=t!jbZJCuh(!P)p#rjA1)QLQ409PY`4=0-0|q$JMNmVlBPYL$q)~g z`&CHIiNeKSC!+sW8^AxPpVBZgujtl}uNduVj6Piq%CW zULmq&X9pCB-F(nlMvf(Lxv1Lf85JXQ0sJUL{G(x_F~pp~Ra#w(!FkS{Qj9RT6+1Kx z%LRHX&ptdFt_l=&sQ-r$Ur^Q1c~AFItE~r6k1?u^u9J74s9pb&gDrtPQ+klDmt~pP z(`B^pd)(gplbi{g9-GajcsFE_Ya7Y-H|K0rr4Ftj9kJSvQl4tWY@$JPQ(jzeL`v`B zAObsR$AO)lGP<~|;5cEwoUs*n1MgU}sPplc`2ul_w)|>wF@+8da|t+nBx)?I)cqwm z_|i9BWfRN;pd`ePj^3ZxS)#-qHL9_w0NYd9Ars7v!z1hml!RkLNlW*^amB{m5xJAL z-Cq-IZT36SDs7L$*;*%G@*?0`p(3@y;NCdDk2#dg-fI!DhJJY*wEXz6WT3#LfVyzA zhnXypitQ$z6iSa_-Wtu9(i+8%f!^EB>+SDB?G9HPRfK3f&RSE@P=X@YoL{mwp`>j; zlF(1xsrR;^iL>jUE-E!WiiB>D&y{_Cq#L3d9+2H&C$Eo%lQOit*rzMw^CLqY|9!}bp^hdwVkADI%QDDrHX9Iz#sl??sGnj-WJ*^RWq;hGmZp-sQ=dMy53Sf!&6jOS4Z!_+&d{dI>26|=pG)XQtxF?PDy)!MObsT**2tKJVU-s zYrl=p!?t2%7cig{3y}%!x*?95m-uq_;?HGF6}!PQ(G7qkCqPefVpuM*XVddo@dW?` z>;)Q|5ZimEDS^WyQgfa%uN`SAX&p-P{UlV9$?G2cgtW`$oj!HON}jXH28%Fm;dG&jsXmPhEbkCMit?ZmxvI zO7a$s59M*I(J%c+eI+eg@1=k-cx-snaL6eA2XR|L7ceg)uHrQLN_8^pU-19Qx11ZL z{uDCWmArr$YpcLXK74s|)Sfj3Gp^ar1x!^*IG>;m&>#)`uPytLa=n`%se`!NdXa7P zc_gP`G&zaWxdtM>W+h46pKFr{6n|M#C>Y9+%4UvDcR&xX*I|&_57W#(k9VlAw?#J% zho7zz<*pv~<68eQ*`UB`0`El6m0B|o->)){_Z*hH`>M$XkMR}lW)Dsr7Q>*Hix)95 za|J;7!0_Pj2J^k9`X9*Y?{BIT8O(eAAWZE}Th2p?kLex8f%%-g!Klj${{WU;PfKb$ zLRW6DC$ojwkU_HNu;~J^Kf(vQJr;rweVu{g8a_Zp!Nu1Sy0gIX) zl&Ca6HFiS2-{5&Ltegdc+N8X4gg0+7i+<%$@!WqX_Oi__-g25)=-g>tuShEV=QnhvHmDG0Q z9%f|gACKFoOSo{=aAzy(wSfYT(j~DCWvNeexH;Db&z;i);BI0vkF}B${Q_6J#%*Xc zY;RJdY0iDugVBf!Hb|(I?m73|d%vH1?zOahUh0b1 zrW1_`SH9HM!Y3BHD|JbQ3fQ}(Xyog_w5s|b^cW`nhVyqT#KZl;=6q-6MA-Yg4$vX9 zFc`7mMW2CKLnMqUsyy`*D89_IvyAQx`}KU7f)TOS(PCVzJE047R3Qv%MQDk%zId8B zT9`JKdmyX$7I{w&3H=kt$0QITHPQ=ptVF5h!Zh@Fc%0qeyI#gz9BlqHRA6z65?j`t6(zrJHSr}d6Y^y!YT?8*%n=>+%g zkf88_iou(9_8Y2*e)-d8qU5X&HnO?y#HgWbOk|~2TSrxnXYoT(TH|sx&e3b2wEWZ2 z*)Ja(Ssgj+0;?Z33l&k8G5#F|K~uV&^;Hg?G~Ngbw|mU!56UFUH~N!ej*9Jm zuakH-7f6ENuG9AYEq_ZU)vN>Oao;8#f3&;%tC0$EMNo#UE7eE~an!=PQFMmr+W0EY zx6xSs;q$K*@lI7`E@mcdvIzsesQFss*m$7ZGZGuM4u?u$pKSL6j2EkzxVSKbXHN3f zo^n2$fddwbEpDdFZT6dHSq=MYR0~o#q!nsmv$Qi^*YQ{7 zcLbaS{rx7Ur>-HsHxL4&FLMO>Kv>uTL!Wn`O#AC<;d9Bz;(QCyqAmx zx3Dm}u26rVty6y}A??##6&09|a#G^N1}M%b@ii$vtcG za>Ec@x&|3n7AcBrD3q3vyfwJi|F&k@jd)gG=q4aJk|_q z0BX(>9beO5qn||d(g$*V$X!+qRIt6PCqpu`fU2<3&X42 zX>+akS_chDn(b91ZL_Ii>mvcQo^sh~_g~4VDEVZagQ}KiTu%^pYI3tp_Tmz2bMsnd zoV8eiX(pWlJg=EC5NYO^ugrA{juc!ai;2aWo)b2T$+eg&8ua6yr^66fv3 zZe~G2=O|*)>S@=>p;#Z$uy5KB^wS>2xU6ftz0^c_vT-HRuNZNe(!Oy7-=JoWO67`C zRr}{a*TN|Na^l;F>y)(hq`e) zHX&G;eLn24sV&T3yX*KHApc_Yl%p@s1OvS7v^Mhc^Inp>kHoa6#;Co#`YXf{0L8%9 z=Jo9&^o}6Th4sO8(A@(JM@NZJjZkEG{xKviN^0Fy`gOj`)M-RU~ zanX1y>^t5TuU`PYd)Is{VnXBIt(Q~WpIb8D8G&A%^vOZ{HVmvnM`hB1Nm@m(}iohk%JFeJDxxi(<@-a%tE(#0uOJmDs_t7IjO7;q$Et=gh{fm zDLa%jG-oX0J+F8?UKt(+l-wS^g+(;7?AUspBfau_cX+R8!T3I#KkI_?{_SEPNaW;C z-VdQE4n#7p%n05VB&Y1>`~6UBW#tDhZ_52>qDLS+HbisTrTn)nV?kafC~CE?Wau<~RYqB-sDh*PC^3HI%JSf2;t(Q-=<4CCBxaiYxNblNXW(S1%6;nL#st@EC*QngX1Cw=0geX!tR+5 z8M&iWzUxiin<+ll*8I-8f(R+a7-u6F7aiv&*Y(MA=9!q(bK7={ zb)9b`_6mMzF%%5EJ?E3zq)Di(!0N$4k9vEklQmv4rwS_bw7yS!O~l_DPh(IX-Gw^9k~H?e2I)c%XyyTgvvN;ZaF0kC7$_M2&;NgO du*ID-aP-6>5~Ec@Xn(Ef%uTEyAB@~l{{g#D!rcG> literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/samsung-sds/index.html b/content/ko/case-studies/samsung-sds/index.html new file mode 100644 index 0000000000..db4aa479ab --- /dev/null +++ b/content/ko/case-studies/samsung-sds/index.html @@ -0,0 +1,4 @@ +--- +title: Samsung SDS +content_url: http://www.nextplatform.com/2016/05/24/samsung-experts-put-kubernetes-paces/ +--- \ No newline at end of file diff --git a/content/ko/case-studies/samsung-sds/sds_logo.png b/content/ko/case-studies/samsung-sds/sds_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..0a172df65d7e152f91c088b71b2a969853faca45 GIT binary patch literal 22930 zcmeI3cUV))_V+hUKxraKuYw@G_YO)gO7As5qy<7p1f_QYLFoblQbd~cj)Js+(xf9u zZ_+!wLC-xW=iGBH_rA~XPx6FhG4ow(W`AbQWY11^h?Dk85h?;NmPwOMBkX?~mV(%MS8;L-x?$*ug2XyPHDTIoUYa|IW$6 z{C6LSi<9lou34C~gKR-|AbV#h+>i5b{UFv3&JIv(hkt1D_v(La7(OP-%D=b$xBa!V z``f0W&Uanm2!2EQx1LZ94+x0;0SM~g;$#lG>k2=VTYq=-&S1+w2Io(a&TIa$?;v;Z zKe(OO{Brx53cp531b(bS(oP^#X9p(@2M1fRUs`a}-z;>oqAa;nol?eMEg$T;u zGfPO?!PdbE-Yt+Aw+Q>+6(e~4%r+r;d#JOiy*Wr;MhsrS1_oOQae%l4xVX%CSvfh( zIaqnj%{W<21q3Wv1vxAQECfvj1-VR3e?nANMs)wT&@v9@F6T4mx6l^Y2SV<`Gx+CvrVeuabLqb-Y{5U*C5WjL6m&l8 zL}~vu?S5a#eqX1~eSXefAyf176-~_id{TfcMA-jS`bW2Z_x#Nb{Qo{{wwmIUH{sI2>bb`fGzmXgYm2M&tvqj zto`>f6**trgy8uA@0;n*ao`kT|F^O~TK=WiIUiSBm!EomE)MvEo%8(lH&?{!zq;!F zXII4Pzq(7UTM@wZ9Z2c%84nzm^7d_{joDjE|R>n^%PWuf>1$Py@SzY;|P7 z@Hq@U&rB{(zTcewTJyJ_x_@|ba`7X2BGmlm`D?HcKG=Vb<*!osX#@O>h5h$amOl^k z@6+M`*!bg+{*MDAhP!M8k;aAgC9Vq*Aqp;WA=0?ezQlDQB1FL@E<_p^+LyR4M1&}~ z#Dz%XLi-Zeg@_OZm$(pVTxeh7x)2ef;1U-ijSKBdTo)oj6kOs$q;a8riR(f{h=NO8 zh%_#=FL7On2vKl}3z5c!_9d z<3jrq*M*1>1(&!GX~}OI#NsLKIx$LZoq_eTnNrM2Lb*T!=I- zv@da8hzL<|i3^d&h4v+`3lSj-E^#5!xX`}Dbs-`|!6hz48W-A^xGqG5D7eIhNaI5L zKjOOj$4!7Bd-$DyZtz?D$SjRO@Y@5a&E*~_1Ar$z0QmX=z`+^(eGvd$H~?VD1OS9y z0sxUijPb{NaOs4+jHHJ9$V!UaBl6jVHlKB2?$P%(U-F1LW9)|G!weuO<~M>J0YBwy z{^U^<#hV~zY2rAEFd9%}EdC@Doja7iKQ?&GBC8T#%F~wm z0?fQ!WpU?(L_YK`t;F$Q-ejFN@8Ix_x9j12&NBIAe#)!g4tGBlN2+NHT#tBjUsGBN zede}9vL7B~+bVdk7$l;^815u3WunfNGANRkna2Yi`>~?V3~j}3FVX!Ti!ZsgMMz=Z z5Q6;8!qBFuTCjFcZZAFaG%-@mTb7PgzU1XqAlVOi?u(~}tk5lsWslKV)lSY?K&m{< z_Yfa^oI>)+2@lyY2-bwu`mG3yeK%ROcmdrCJ4K+2K|&;CoJ|lmn@cJ@LOP~T^i^xT zcp0MYqFz+Y+&n|dgALGO;!9vKqR9)_4ZY4nhi0$d0yhu`mQcSfCVX3(cS}*HjGb^RGILI=v~a7JP}-+K2?Bk$;AX!MS=< zl+B**1dHqo<>HZG1bd6ou2?HDa(11BfEHt1ch65n`iB1RUv4+s1H-SsYY%rZ=>#TR4S2_KDk?g0L=+OYgb zJCS{N2OlAuzJy^!@}S4t+8;xpeXS>NLrk#zCE{Pn70ChIJ&T#W)WGeV{jdD}V#P+n zu}}g#*F*tJTC-kh?3?NDc3D@OkXqww}9j5n_X~zsPUL$9bMOT61EL@l(MEPT*|v= z(LPhQC^SPI?HY6qD?=o+?#8}t%?=e<7K7%j8SR*%)1Xw)p$Vy(*vG58m}E=b{lL%9D2SIp zW#Y$&#E2tdg)J`1%7@4pua;&%uHDZ{;M0wf2rWsu(aDt#1yqurCxM}Y&kdsNz7!eY zae>`cBHQ9;-p_-Ep>?hwb`~|!Dh!2MEwkKeE5QS>>=NXj7V%VgrnndPB&?QK7vMK4 zJc5x&b(_aFRVhb@LJbbYWFKdrJY58ti1gUB!_<=bLjC#dbzmssr0b4)sXp^t^hu=X zvlb#gof_|kaN>3Xh*5n};d9O7U~OnkV24VP=5)Z0PK9*HHv2`It;3_09A(UrCZuALIl7-#1q>J zJGl@t`_Uq^z34d!sR1TM?_{KEqnK3yPtm%QZ_wfEsAo^Z#^EkC-BpzsA(U5~zId8L zNY=?n{)^-iirPH*im9A@Z8I)1_(A{-NQon~02VMlt{W6NLQLmgjd9N;t~oJ^K5=jO ztdM;xoshue=IK&bv>%sh3EtB8kn6bo=ssI*Dc7_IsdHQGkyz5!tPeG%I=f5OkKct+ zo}`Uyi`K@-#{AHcJ)6C#1d#!R36O@V~?3xA?AhOY$O#!$qL zo3R$J{Tp^-Ew2VK1OC(5dXb8N=|F~|H{lg%(`s)L44^nEx3X2pjgMQ|v$dbQH`t0S zeHg@COH_w_f1ed(HMaS!nI2sNNJ560YVVTUra{+BM1zFA?MH9$D-G`+dCaukq|I)o zji6jXft1nfs1Ko-SU>5TEqMx603`bMCt6a5Dce4z3+l2h!)_UdBl-5RHPKSoV0BgP z#YP;BimG%vyb}d2D{kFZX1w=B?q$`YtRTiWZ?G``aeax4s<3aL>K%mz7CZM4lwgK| zP^q4|7jvv0mM~)~(Hwz3q&N~xE_^1M%8k&CPRgW|3_J#1qz{-lw32k$hFjkg52D_# zEJZe(BBj^0Kz5amP7`6BR63`PHltFXzb`mVhE=3x>kuZXlRQ{JRvRT28M%H%?d38{ z2UKc?EeXw63i1O|F(757a_6?c#t<4^3~5OJOVJjs(W!>bA!d8yqAFoLDlx{bff|(x zAr=Gkz}cC(vZ;+%7GGz*u|wyB?`h>-70^_EZgSH|g`VZyR5a)9L83fIbqlyrvg2 zn2ngcu=r}fjVygqhojSl$hAsAoEAxXWC<-fRWIw|_oB5b|e9zP*>u;Jv$+^KX0pbL@8aIrH5hDpk`*ThPxJFiSZ)Kvpp4yTTDhYUHmM$Hv<&tj{R zPf^Q*8{ENL*>ABSx=8U_7S9sV%G-U6Jod-#WX&tx@w7bE01VT=-|`vB3NXw1n7S)b zTO~S17;b~{8F%xI<^xCK^>oc_IJC8r?lb+FqwC}HQitQi z@2nxBj_75A5`!z?1ef_X>V6hh-Y(OItIWs|PrkI@f_7dTi+)*sB&Y42Nu@OmuBax6 zQPWiw!BpW|iO0GdYUAm~vRJ~%mY>|ey=L8PS1Q2bBv2T5Lod7Up{BJe56KWQG0IFF zs6K=IMeBb0Vd~r*C2^)FCM$QRpc5p1fcCw!KklH5WX3ZL;96EPUK}Tf3S%555`A9b z6C_NE&oc75yM$J_N8bu)T0%w3CN-`K>b^ivUZ6&9>}I!rwAd{QX_r_pkaoxg>qW$; zCK6<$mY^mi`g|c_d_qS;Lqi-SBXP&sa**v4s!-RBJ4ohpXb}oJDzBC}xQr7dqK{sF zCg#B=f5u4A&N9GRLNp@@Yl^6{#1;xNB;gfUZe9ctDKS%O#?vcKewnpp$g{Cu4))u0 zCL3-zje3c8h)>-PcBR);mdnyOxQR}`V3F0Ws# zZlpi_o%>nwtcDBm0~?vHZ{ib^G})^!3CGUEaJ8uIij>ylcd#ZBfuOs^HP2&e!$Y!k%>zLxTs%2W5X;CR zCJf?FY;DosqB-id%}cx6nQEJG^JnU|(v(*`YNIXH%UO{k%{pjeLXG7@?BwyG5Of z9B-&{TXd^Qsdz?POs~86(Y*(;79!rfiXFH1goY<-^)sHmF^etbS{WvZuVrZ~k{Wu3 z-1F*@n@3??<2+ZMQ+2vt{`HixKYNq4l+uo;(QrxJr6MMnS-IACF?%5iKj%mH?oOabIxG9f!OX9dq z=0ze*U|BJzI~Sy2q2Rx9NVG&p*)~lMF!*mP>JWrT9eOPiQfm3`pK8rx?4@TrP? z^*xQ6vxMn<1AV@O#tZO=?~mad=<^*kd~f}0^NUc$@cN3%4mP^n19LB%8qyIn>ROgb zLCfH%n8%88JSn_xcOxikNL=Z;TL$&B34`g7qb(kaa$og^O?+JX_GXYGk}Mx!l(Wy4 zj_*!PY>oVp9w^}^*X_)ddBy8m7aJAEz1UL-vTqN_P0#`mnt3~ZJ;E`cWfnAoROTXk zgOtISR`EbA>3$D!WbSynfUj-ExM{4H&8Mg`R2x#MXn(D#(kQZ+S%LaVW>sT(XaTFi zINr7_ewC3v2y9t6RDgo}v472EOo#Cb9?Z0~LojzvV38R=kIXZYCm>T^l)qC;Gk(!k z??I4>$>0raa=i9X!|uI?S{Exj23k_apmC?xugz!`DSAdzYbVde$9kPlV$+(KiYxg{ zEl#IpE!$(NQpT^XQ$1Q@udzt#lB|SH)UJo?<`9t0kDs*;p6ME}juILUadX+`r-|gy zwGk>V?A6!XeJe>^OU2;9Ntm;!v2S>^vPu+n6;p( zk?q~QYHJS5GFM`w#GMa2>Q<^uN;A)VIq*m$bKM$h57^TN0=zPM1zDW#tF)eor@D8q zqywj$y5r;Q_Rnl@A{TP1NE*0$c!DMz9&@GFHro9d_(E895YU}wYZDwa-HXJ`%Dyco z<}-4j9;S?e{#~mEf)|YLSF9%M-ks{|=b&uP>{byd#p$C;w-DuuIvwr^ zM4f{6o{HP8<9tX?fH`Ug6y2dPjP@e3)hn$alwgStN5%hU!j=8-HWRsBx;rD@G7eo6 zCi?Q#+s&ue9K`DnaZdL(NoR^=VgfwP(j{05mjHqJirS)$sP#vb*lk;(+_F)u{xUcN zWyZUsgf)@QIo{aaUXJTFqeh064CLXUIm__3Ouhkp?{~(wyG9IygpuepSn-3=z097d z(c%wDr&}vRr`~3vvwK);a%3R&x;$x-k) zv!htFG{*DS&cNbBUVB^0d$HSX`|+J`FTqJrxMQO^=V1 z_523lWpi?Ei2O(yQ0tOwjVZ-Uu=(Glrz1W-{Do!3EE%4+#>v~~63gk8Yk9OES zoljF=x^`64Di+=5(~LAR;?VS3$aX7mbsCOU(KeQo2Kk)>n*;>jF>dIvmSubub~v26 z9#<91O-6UqiImwX3^jEGZ#VFNSNLdm5dQ! z5g;S1FRfV4_cET_Zn;DmT?kJzeR(oT=nYSgo#sM7pCWxwgdVG`eoGzu=#Ljo5L(hc zAm&?HB-@^@h3DtR*-WNu_p}K2MRqm9+nPLzdpHZe3PY7nDJH3~Q;;6D)bjxpqK<}E zfKy$0hQW(&T0>XMgWzb7OeU!MxFz}>fMFDCw78(V|qO1Ybk>)KhX_*X-)( zgtE~cBrLHQjPDR=n83kv4T~kw zpdyF9I~w2G^KQD8PC7c%sK@KlEM~p&g`hZ<{4V~R(M}I1`pD+BCJr+(d!2pB->lVc zHmS1h&*tpdnZEQ|{xRWPKIDVX0r)4}zmiVi&k#FO+ourRnTHk&pZBGA2&kv}zXto_ z;m{&+JocVT>6sUQKi4`xIJy5lk!a%Rcz@inX}u<6iQz@$y$9$`@pY`wb=^Y;+;oKl*#Jy98~B@tNpK9`$|Z_9^~8-{TNYUxu_2k3Q(j zCk%3jb>#HatsQBG#yx!&S-7Aa<+GbcwZbp9xjJB&LUCR0B_|(MW5Uce(@c}&(v-K} z{4vmlvE`C*<{spx`8H3jRXauhn37Sk$8we;xqf~>`m-ziMO;^3S5PL3R@0&Ar8ndJVPJp)!>$ZT27gA~ z-I!mAv(46;K)EW zxz|Y$LrDn5xgzB$YXCak%@nEBKW1;+Ol%+U=ee#}3 z9=rCL`ACukTtLI~d$;Jl!Pk*fL`P5K z;v(|ze;;6E=!8*iq&rRV;JX$tg+D|^hg2WFgE2f=@z^`DC@GLDx^uKwQD3{SMn#pp zH@@v$(G{(bUJ0vY`P>O(43csia$z0+jD=QbC^Y*#hd$S)qkxPZcv0>_O12FqIU1aziM?H~4Txt>}KE1x0 zMtdIyKomj@(`84WHazlXz_;xxDyVt)#`A~lOs*8ACy3diq^_FV4(I9ixIiljii9(c z8lH;#NYU6>?#G4>*DJ^y1MpCcugI<))v$vIq&^PJd~kW#g#rt2|}X4KvDSXNwL zn@>)2^Bl&7v7EJta%U!*d}thdx6Xh25z(d2ihrI-j6tm{qo+8^(B&`A~oK~Gt!!3 z>vCx(4!dok2YT#K)+UgnarJJ0g_z*>V2s~8c7E3G?G6z+041G3BCyyEZyr7JFajm9 zI`cDx+`p3IW|$1g4C)-^H=&sWvH60Kk7_O)S6Y>{uRSO9#ImYxfYst~lQMEmi2Kk3 zdMCvJVYsz*-+P-iU0O5y?h*QOLfG0Zp#(F`X6jvJd*8CJSd%*Qm3N`P%BHWi-e{_b zCpW0AY7#Kngrs@obbBB%u}lwvm2~I4ZenUJR_*N$vntprnjAH`;>EAJgtP3_*OW_Q z)4$rvx5BPlE@7H^?s7B&tA0=N-1h6xRo|D=JqGh?7DA>Ftv*h1V5v-*G=9KlhVe;W zUX}-mAX;if zN`GR>W=E>#DUlq}Hj-A`!iICK=a~SNp#DW;&fg$Lnu`btIcVXx#$aZa&1aAD;9q5XAyG2{j(L zTrp{eOa>|9P^&1)x(5+>0{-SqW)B78m6p0z%Z`tauj8_O{Y2Z&U^H&l>hnCD#1Jwr zNC+^YTb&$5F^n~ss5ev9E$6ye3hRX^rHCxd6KRoN5lB|Mt}HV>M?U4&3cr+8t#4C9 zeSJyw#1diP=QDjD5xq?%Va^^LT*HncySx+0G z%StnM7w*I=uqEBULY>PcgAZX$s^GIVGQK8d63s$VE?bx+VyeT}sG9r3Pz3r_IBYLK z=oY_tNyo~jGRNM^H!Bqu0_syc><$-JuVb&XgVQJDveo#cFX;8OrfQGF)2-4Tya7M5 z(9PbPE~b3v^5AZ{V2sqWnfGVYUpUd1JS%0)99@<^)ppKnN=kX8qBNqZqYoJ9ebnHy zvrWxV1BkjA$uw77yb9QY&>F{IXk&MDy3dzZ@2Vw3*IaZ|DdM2rRkxhDBSI;i=CjOY zabccYr6ZYdX;O+=f@a}UW+++zTM08j5ODgVfW|KKS|AQz#a+kYxS+z+%11V_w&~Ta zfrM4z6|SQnGwE2(%;{dUnU-1g6e_+oMa;nM(AZkwXwS##c>l%cii1Y+{n&%0kSd4$T~z>^9BQ{h@v*N7Y5jc| zogK5qmo#>qqOz)zdQ-j$w_6^E9MrOhkC}2A&Z-lF*zr(Znroe9(0>hw9%pOioC zh8=}2NyVJj?_(lOSLB-)JS+NdzqbD$eiiWR%l>_g^GhFpeDQBhz2t)qd`QK7z9kBf S!G9G2$lq0wDU>pK`u_ku*xgnD literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/sap/index.html b/content/ko/case-studies/sap/index.html new file mode 100644 index 0000000000..856dc8be9d --- /dev/null +++ b/content/ko/case-studies/sap/index.html @@ -0,0 +1,4 @@ +--- +title: SAP +content_url: https://youtu.be/4gyeixJLabo +--- \ No newline at end of file diff --git a/content/ko/case-studies/sap/sap_logo.png b/content/ko/case-studies/sap/sap_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..681a3fe5cff8c390ad8ff822ba255ba397ba65e0 GIT binary patch literal 20435 zcmeI4c{r5q+sDU}h9rAL43#CzjA6*0HDo7Sg<%F`8_d|Vq!8IE`@Szp)=09ICD|%U zD!a(Oq-4uGs;B1Z>G{p?eUJCg8D{2mUEl9{-uLHP?m6Z-ZehCGXAV*wp#lH^2i4V- z^>Ob5xMu*66!-d3w+W7WqjXU-#Q*>Y81|kdfaFvb0N|7`%FqOBqNOQijdl`-BhXez zVJ{~aoEiX-k@s?eTRR}JAS+tAv_ z!CDf*EiXqU<0XYl;Dp4&L0(Rd&KM~#S?+InrEu|HGngCnO@eihC zjKo;GpWgFWd_4j+N!+-YfL;@#DOlTz(dE#{6K1qX_nbyMRT6A>com5Y|6E zF79rQ-wq974MsX5osiC049+L=N1uxw8jHr*q5q}i&+dOMj2jazt)I33c)U(de=Lo` zs(9cK{6P9gC&tj*1qs$iV$kkx)<_i(+^HP<)7fKDHopevZ<+Rb{`J_AUZ{Vu?e%=O zeVYp3M@R;Du2M>FNH`YlW{5^R%6*?@x_==ADJkvEDwr?~#HR(fMmg`1<(C2fZu>Rn zzcM0~;aH^H-kcPINaB7CMZ~3`Fe#X*5JW->0>Ml2UHMxMEi?jU!yOh< zPE-c`r)E6Yx7jA8?u^00ovo4T%5t~{VH64>B?`BOAWuo05`sZb*$9bQNx+39B}A=- zkPsUyh`1;cCLsa)22o23-~Nx(%4lo%y%qB#HR6An8qfZZ)H-e`+-?VV#MA5@{r|Sa zAJY7zlo|?yo1@-%a~C&*zwKuRNY}r&{?*}#`nE5*z}+y&y;Ud6{nxVlxsm_%$>!@=`xyN-YyUZ>GJBhw6mCA?jt%~8 z97JTm|7iPF`Mcg^Z(JSSzv(5Q5ZnVSviJJKir@Xa)$~7G@waBCQ|H#4Ckf0=&o`J>bHms12P z@uS=~CtlAF=l8+FyRd&7%kQnYj}5p_Ea0CXS^hrEKbOP*bMWhw{+|=$ry~}@r$I0# z;v$F$UqHl#PlI4g#6=JhzJQ1ep9aC0h>IX1d;t*`J`I8~5f?#3_yQs>d>RB}A})f6 z@C8I%_%sN{L|gGL5f?rUg7JUEMfK|@K%_J7&OcAwEq;~9tio}(2ZF5C^tAv0 zpA!H;;1vL1V;A>60|2-~0DxIb06;1c062`kY0;{Rd*@VFRxtE>GoR*piLrgKJ!8!?|_! zE*gleG3)A46&3OY8^$W0JgLFOlot~ouCJ!TpPQY{sQ)iro6T2galIF<=@fCsJtxk% z=foNJoH!%S`MvA5Fi#IKn57Jh(aIU}EX6ESl8^xzNP+hjS6^EK;2{4c=0_!_Rr6ft zNyjqMhHL38X&;?V<*)B8PRa^>NbkT)7I(8Qbm+pq zd)|Y5^oeEh#+{rung$sb)^eRfrB8I|2sR?$&vQ{@gmTc-@B@BA3|zBIHI?Q@3cPC8 zszLrX${#!SRUynM&DM{(o4~a0sirrx7^Xc4!qslu;vMnzrTL_723c)C8!7bZJmq68o9#&or%$520tYm#S-kPt^ikj+1{QB&!MK=6n5Lr!{ zlTij04JwCXK_P?ux&GRc$TOcqH`>n8lHOb$2<6%BQZss12CG0PIJk)GOq){NZaK%1 z$+`YSw0QK+ojzmTqA0yGT?+r^{#n0IZ-W)|(@kCHR#B2|zT*_D+Ew+mR{>;vRJfnl zJ9d;HfF^(p1c-Ot0rlG?D6T9=nr-UK9B6`&BtBNo^X%w6cbY@YS8Qo*%dg5DBD;Qe zA~J3TBaknc;zP!wpE{;7X+L*=H%Wi7 zA3tlXeX)?g{?fgQe7bpVOF%kVl%5FJ*JuU?#5j4p9)RU40LXJUlG%s7o5qhUgk0N_ zH3)#apP2dd${lA~OHyjm2of&t4wmEl*y$}v69J)tV}VKy7brwqa1)LbsqVcp&=;@W z1HIGP!CJ1{<2xHeGSP1?>&zJ^?x#GGT6I6Mj5h6xJ=?r(r^BJt+HLQ#jneR^XP@3y zxT!E2XzeF0XvUEqBzc`bVCP{jOUmuy(?AHd%)_G5`h_zfW!=!s!E&FCt0x#LFZpO( zP#Wo>lU?s%icnEU*~Qz>CW8%O>HMX_CpQ*#Q7>D=&1jE;mYYPJ%HE&!4_gmB&IOv7 zHr5n@2DRmsWI9#QEbpsY@9mll3~$MdaM~7jzxH)Rw+1dstxoZcQvFbuLYD z2KcV8B!`Jvc~PRx#YmTHG-fdgw$`*e^r>$~Qi1!! zvFl-+8MjuquiT3M>NMEdxT|-;kdM#g6ZSd}Vt<#DvW=CEnaruCJ$ZlIX`^R@T?y|a4`~lp zx!F5j&6}ugQ!%lLP`?GCm>OH}KV}-6Sw1DC+*u-Wrkpv`Q9_q_jUzvaU1FQA< zO6)sh9q^HuBL}ps*d8RBu_OA&={mu?pR4n3L8W`;V&>8nJ=3@~bha;zD^-SgSiM?r zEGy2hV7>sC=5X>>H$NpO%iYhe$y3>$ccq(}r7nE-pw;x7FfKBB57}D2oB-0-d6v@9 z^Y#ea-s%X6-LcQD-ZRfs5HaNpqyWo=Rn@#5|1qt!#FZYj#_+H|xn|H)R;}%ex2`PI zKAB|@m6E1B-g@1-+c`h#WKrnCV_2o);}Vdy)B3v{G%)Mcu)n(Z%oysrTOD>{-t@YZo|;{5NbaCol-bTL z*U^}u%+&i zwJ7XHz@>@ewpppNqr7o$*qORM)Tsvvngn_Eao zKtNi!;#E`YzTMT%u(0Pot%*_(dkxTjOqnlu$r10n+ikj7-=3foix7&9%YD6^=u)3` zW7q$D!+N7F=+XG$+K&4^d=Ozdi+i)JIs;CFDG7aZ6J)krEt5mnr<~hF>^fTJJs2_L z&-(p~Tx~T@DvZhuF$R(LKgr^fG0_CM2%Tt7bR6{f(x3MxsDL@-t)|)j{DKyfV}gk9;9~9H`jE-agC30~6sEl%t=g`BdmwU`l_gEcpoA zJSP()fA;)XtQR|XhkA#Ix5Q`rGm=^|(hgmK7{uZICF%TB`G=a$3X;LO&d|g)G|VOT zTz(GcDyK85F;>204Z~f{yyu?Zjm*^JQzlmq6)dJfSLw5Cbi9!?-5Jkn9)4qKydY!E z3ZXRjFR!tH?oMGV)hp=>QDxQ(9B;fQ_6N33(kYz=0S~iogeFO1Mh64@{8Tfx)KW}k z8x)!|V?4jwPR)z#WM1!NVXXL2C*|vxqa;75l_cBfzrq`^-P$sF-^GqOYjBJsLCM@h zv7H(I=ylgw-Whj>yS~2Y_i;QWyHjh?Y_ahG8quxtYC`0b9Ez-@EJ4 zH`IGKRgRl?AA;H=RN!5IsETQrW<;up*-{aDXO(ezTYfzi#SA-plf@}TEccMwO^sl- zHQ)WvkO%n+S2jQR&bez@wZ^F$vn9~ls%$qWn^vl>ht8={Y;01V9u2=PApWqX)Q>fn zM|D^N^JI8>aocX;-OA(5ASKe!d71=|O1`I+?fI=C4k5E*;?h)9y-Gj}w+8!>u3HW911uDEo5 zq+0OYS=Dei*h;a)VQS78h_^|G$F%F)!ml-vv%A{<&(Y>5PX#PWmnELxF}-YYv%vW1 zlCWU$eZvE!`y0J$*JTYmnUuenr|#s<^qJc=cHtL&ehyAJX5{f3o97AP3b>lFt7)!9=^%LHUvxnqNM5WO#rBRb& z@)8bMkJadmFCRYcwBBuSQ18$@d!N6uphs_MDg&TSa+xfB$h8b{v?1vwiycj6vQ4zs z?i`y%VADS8Zd+P2%)o1p3&D3r2jpZa*ykJ%_<#BGDJ)aZ9?>SLbC4EN?*6IUAYgs5 zO=wG+r788@F~|1J4ey*67sj$_yuW^-PK#Oz;hCv?oCjJ40N8!!9g%vdJmC*Bmh`DR z9l(Z!QX#@`>Zu>5z77g$RS!O04vCRu1(R}zgYkPLasrFDqtexXTh7PMJU##ys6PeLgPqHZUF4*n&n~ue!TsDl8Uu zDCKC>eQgL8KjuY0r~ByMvQ5oW=8g|X> zM*`DYM_*g##qmD9?H*+olOn8aroffPwo=OXrNP$IdC-HU>(QBjSv!U#{nzNa$VMUQ3}5WxwMtc3^(Tle)88{gwL1!Lg6L2YSh9 zV;EzYVh}M*mo{eQskCR=UzXY$4rdO7G}7!2^u`Xn+f4CH5BMkm`S^CAH%+EW0!!w4 zF(;99dQCO@u!Wre_&DbLXJl?$H>f6FP|;-mrfnEuDrRx|f^3Qh_e z!o;KzQONN}LP_d39``YjN=_ubtvMRfQx-f8w(xV`Dv#t`tTS_E63mYt=&42OQeh8q z=`ogibNO`nX5CfjPgOIyca&cMq(LW=r86usvv^+GX_M)B)~Vyc)_T{3?+M+$TiM?2 z=dmLzv1|Xi>~pHNzJ{>*Mn&PJ_J&I727c6?BF`SD)s6?X5_3n;_6Hhy_qR&hgjq%# zB`>9rT?6!z_P%-r8>w8hNb1l#R_WuR!LLdB_Q+%UtJ#mcL{&f6T=z+b@=WRkjQFms zp0lKb(^|Sbar3)n=$*Hsc+OA#{keJS;xS3cfzz3FTz2eKH+X}p%V@8M6$=)VFch?9 zw;Kxb(AcM}vp2prWvp&bN=x<2zN5On!wvSkbbr2Ih4#LYK=1wOxCs*=9e^u_351vq z@t&w-4YF)WnwWnv>&q+D)$Qfe*Q@A}F%diuq;veO@dN&+v;3JpAG~ zx|Dp6f$u8AYp3Ym#cJS@5XI@+7qk1pLh$mD<7Vug)Ed`0W0^awa%rnfj&(%FZM9!d(Fw-`cYKt1i!j znXb3rHHSVB?JcpiOuWc8rB5E&C#>955j`RquOR;}Iq+&z^-gPdc2(o@_E)96p3g^J z1R>P4)KCvUy;f-EXo;`3?B;}CS|x%@5yT(SwCL~Fd~MAud~YAj=F1wJDuiHcktX_j zzN3>TcEs`$91f)wkyiqEI0Sjh5}RAdCeqihOoNmG*C{n+5`&NN3PdqlEb?6%-5d`S z#nkP}WN_OFQqi^d4-B4>U2aR2_fHJ?z`khW(p0z6jyBzJ?yOQbd*hd@dD~$vy^(}O zaq%mnH>D+TXl1h}o2CXx$(JbVno*J=!F}xVBU{MQRX`ba{EEB>NB#Tenhlhb6EPz`Ix=;i=K2;E+Tj<@ z)drbf279@Ys5~o_g#nsCK)S#fvXS_sr+j)C;Ya(y&RCllWf5PxCJS>DT~bs=B})g| zgikUVAJI-N-h5c*xFunGDB~e{1*wriZudjfTRPGw0)jzR6r|xa8@H~U7{)yG5$qdn zk1n@HlWE%nHo4Qb zPdFZg?bz;I2+wThSGLz`Wu~EpjKXVV-nwKSj!?tipqjF1K&k0&_?jnVCkkx`h-*Qx3AAODVYi?m>S9)J1Kem{iXNk zuIegQbz6<|2e&bBy@`H@-jT@2<5Y7=Dy|~E5z6{%TJwrZw902k=DxcJG52Vjvgft~`+k(hrwxOo{!V!TCxHk5xwPD2e4i#l(5|wyQ@2f;Uz1Ucb`iEO9dDabb=6ZT>--+_9ZA3@iNtMNZ{G1cn zO`rD+!sIF~vSz_NjNZ3ssk2Zyc~rk9(|kj!1VnjAbQ*mjrew&;+|Vp8o~5-iLdoc9 z@$6afK%26OmJCKIgJb`70sdeLH|?s+K)cvEPIq0~SCNnOGpGbl0R`R)0XET}?o3XeD8a8^EH@!1_WqhbYWLC7c zbbSmO6;`q6X9G7{0*sBY^6MebxqIl*k$cQuYtR8VBG+#~RW45;VOO@=^{7fC*>8@U zDTe_pJ%!$ED!pr@q@BFi7sqmWeETpdX`s4;bGrBDRD_JbZ*F^wbz^VXHP0I^b9&I~ zH(eQbQ*NcRd(pw7kmmj_FTxX6F`LSv$zOB@GLmV_y&hM$Ge>T7UAc5{@&F5*96h=^ z;J~I5muzHfLS0q26QiLMW&7zVsQ*ZH4vf6O=W~3Tc@9q?B&#C9(#4_^fTU%E&>?UDf?2ohtH T@J8U?zui+;(N?~zXc_cBN?8%x literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/sap/sap_small.png b/content/ko/case-studies/sap/sap_small.png new file mode 100644 index 0000000000000000000000000000000000000000..ada89de7595e59a5496285cf74ce56fd12d30ad4 GIT binary patch literal 4879 zcmV+q6Y%VbP)SFtF_No zS6};VeXEWwsC8A?I^r?qsBLNjfLw+&0s^~!lFa}$u zPd#i03BiQ&oCiH#<&Kzc8q#3P^r?sK?jRG&bAFZ*^xkfmwgI3Fs!*SHST-^+X;kk0 zexK(@re&YL^v@MDSg>tmfirP*?)@nN@7-rB1^&5W1_!o{04hi2JrD|Lcbledb#KzU zVuS$O2FnK1I_(orLlPiM?vEt7?VWx`-p=WN5(xp~e8)&Irs(*L^YJ7MmW?EYGkJ9G zeIcLcF2l4CJ=|Pk#Rvh02~iz{K=OOY%L$Pl^y2fVfI~cP($pT~SW5&w8g~^ZB=D-B z$bD0i<3L0w=~EE`e41+nJv9>o3It@u;!(NthD*r^dCysl0A5XI`@s`D_QD4A zxCKhLv4nt_&YWo_ESz4BV>`VXzGd6Q;wEk37>!L~4mY&2v#yCl4W}RyKVw)I61I&b z2_}!u`$;I^`>}2$&r-T6#(}U-Amr!ki>tW#s!E3Eq~cK%3z?S{WXy;&-`k@G4b5TR z-h7ZJ-`K{Q&4-9vwrh!JLkNf&%&aVA`O?~?LK>FM#_dO0yWt~HJm)ruo79ZT@1idOjuS!lvfEnWp`2Lrwd1U1W{QQ}9gu`+0 zYXjeZObbPpt*UYP%R_$8eTHGu`-a&WE(-xso$5=9c>ew`3|J}MApl1}b8DpQ!xRGI zI(Y?|%$`!%r?D1Qmyntfbc2wKjGT-BKe=%x&)vJ2oU9PU2S|=sHku+E)5aJ4DAn(| z&ouikyMzGZI>Uyf^1Is?Qji@Qm=HZ695?vhuil`k;Z!%5;6O}g@yrVHv-&pQxT0*P zj?M$a>Wjf$5X_re#N$6&gsQ6jHweo{l7w@~xPs*=LC;U){ZI;kgyn!?^TX?>Q&y1C zzsY;SlYiUJo9p(1zuUfGSV%!1^QueF@k+^X!`yL+FJ2^NT~SrUx2~Du1_gZ)R^Vpz zvSigx9C2IFr#)brcE80_h#8a>4&kcW(q0?yJlw>tgC}TgjUvessXmR&6d!q6A&T?T zkj@hPP-7bp{^7ljS){H$I%ZHkxsXfBbNVIh?8!w89hOEzV{6y_3VPiGbkkz@;U?my zjVuZL9);Y@AVHtj>%7aZoXBrp+e}+)6trGaKf^*(WW9RA@E->Jo}cNui6r&kY7}gn zspIk)F(f5v!$)8IkOv=M&Hj^ZZa6IfMZ)9JC`b=bR+!D4sv;K8tROEd#Lu5y&+a`( zAvMr7FR&euC9a%NPI}PWFJZINd|X>o#-q=?2b$Wie50Wy!ZmmPk=EugvZmnI6biCK z%&95mmTRXFN}`~$f($-8aRh(*>p$G^Khd<7jVjAV^~4cBNbzeA#f^dQhYFIQq#!eC zqhn2Bmj8McjrGlt;s>R}Y8;2SZnC$oiMHg6C@st8@X2=K&F${PwH)>xI>B3i-^YfX$9QV_ zBD4gdv8G6jEz99guWxk?t0$Lc*~p4)PoFsAwvbPIIA#tkO6-J$5cs{xmuFhg!;DFV zZo+6-U1=CqcC+%34?HRdjyChZFK#3nj-Q(h0vIOMV}~a3q!f$TBk~_a~IYTo7&>C?!Lx--`BMH|B`?PZt%1+2q(oH-PH`%zi zA>sA(pqJUxiUD!%L+UHe?k2krd)>~XNxOtP*?{g-8aA4$#HLRg`Mngs=Mn1yh4T`& z&Hf|Jy*5k_dbsz-nY{7v_1t&MTqafIy4+Y>6m-K4CzJA!0(67o!ffVM6(wx8>);9Q z{_Wd@V~KNzc~!-vWd_~VdwTp{Zy_nPo>uo+IAecj-K1%wsd9YAq>KXN^9KD2~?TT;uY;v75zh{RkaB|_BT3)3qI_|$|= z)^0h(>i2fD{m{vTZKhV_GIeZ$YZ#sA!{RypaHWnf0x(R3Dl>c1$fSLvanm)v&VHta zqRPgll_S5C67W1~^fOQC4H90Bo%7C?0wmU*Tjn7n$LbA z$O`%3R1{3Rs}<`eT1JpXmzF0T_b(d`(B9n6nk`2Xwh;mr)|U3`h{1Lo+QK@7qMh}->p_uO z_5MC?UN{lIr{}=rnwnDX9hyo*Q~Mb!P6CgjFu7zXZ7m@rO(w;wQJSC5oa$1(@R^FF zedC73mR2Bf0hA*Jy5ucCfS(^J+`+CP~o#ylyWij<rLAAIDx zK#$6)#x~yFn%JEIeChJhZWL(ue)=d&f+1;shGhiENe_?_>YGvk9)D#sEp<((L0{~% zQ;Kd1`n2D;Cyy5oXF?GN2zWVI-^}Ok{v%($X9X{>+e@r3r-BJ^#ndA1zTr|ZEM`q9 zqBJ)xVcP@8PO)j{Q8#?;a7}+&y))_ao?Ma3*y5oOi(fRZZ1v_t{OZ~D_|rqD=1eKN zIq1`#xRB9TBAgo|dNq(FR=)8M7X9EE=HK=-|Fv=h`|5fRse8gV7EPvfbbhZ~+&kOr z*mtz4gI9Ci>zi8-bFkr5!aivM51*eo%3Wm_;WYUDo%MX_o)yF_JA8R{$t@wj_WO@R zDbUl=vg2}Dr1-H+i#Olj$D12JVtGc8DP=iaaY+$Z)s`_LClxhO5_uLvK9$?PQp3#g z1qs_Yz+3D0L30>_zK*kdL9{-~>+kR9Ul&a3v7Z3uO)KVsj3CB9&rEuNZrVKi)(*b& z(5swijl>qt8go;?qdln`R=?Qr^FeYhM(H?OwH=yIMR@()J-ohV7kB?|4GSh0a^uyL znbS*{39#&n@yN;U$Z@!S{seM{rgFHkjksZw;`dOLlS<9l#0mu%TbRw1G5NHghs*86 zq2sN5xVwShy|Rht-`s}g(^{{ZHTLF!M|~<;;l#zjxr#B*;=(%JgCszbU0w`9IHL2+ ztDAWK?HxRF`$GO@{)D8HBvlF^gkZ*m;mnvYyx%wA*A!+|m(qADa=HMfCq#9V_qNs% zGfa>K!hyJH(Hhn{*4)mPJq>Kyd6aDjPe3@1KReVix4QJ}K23SnI1iS1py;X?%O;TO zM~~_lZBdu(>)edtOzoB`K}Lvp<0+na=|ir&Y&1b{;%xq6?|0OB)mgX+dezB`0GKh_G~~YMW>Jp5zDo3zt;{l$3 zXAhfq)>GHmN?Xjpa2!0cASd8uR!u2C{7Oxilgq(?rpS!zg|QF)0q7<{ zzqe^#b?GvXD!=e?MPG@aoF?Ra8h0$I>2mS1%8^`FIr6k0Zi^aNwv8ssWT&}D%AW}6 z7Quk`h%`%oprD&%=B9J&;;N5* z@>kaH=W9P%$?#$6Oe!8i$?!}v(*h{6ARafVKhehagD3g>)Goc{>2L$WEdua zfVXKuZRuBi9_6KvGf(La3cAVssU=J*9&*73HEoBMwisEdzNGC|Z#hJ`HO7u;13Pxt zcdfGQi~wafRA0OG2z!n+GpXbP$!VCRhJ457RhKUFd6id9)5f+PB*{HR?L1DfZRa!< zXLk^wmgC@bT(>$-Q5`t_JI4kax1Zfnv)ow~>(~}H0>`m&9B^+@x-i67o0Bt0}oL;MHD@#SI)FCo_PqTP&MfhR>toI9&_W9j7x$ zbGrQQ^ye-gx)rxOo&P27D{z&bmEuD|mN|B^onJn)j{kmn6LTsHSvqG7)noI?NcH#0 zbwqWOiNXY(0FE_PB9w;a*R0099qzk!G%2zJl~%&#!DG zf+PuABRVZDVNx@L^;beHMoDS~cS%0hO9UvSV zSapnPk)D@vWWn_EuX;Vosz}VhrzuDR#*EBn|Iuc$(*ra#xAWc4R}u1QgQ}RG*=2AK zpgOA^9S1B2)3h)<6(Ks_!t$uDZ^AjVKWH7bTfL#b_bq@>|Ef2^tS6n{X zZ!PO4nHj-@3u?-)_h`zRwunw{W{|m)3y~y&EKBK4ZPD)?Zw?0oUhTLLj?)^|7rwWx z?)OE*($-q`;L$RlT*c@nnHhnDi!QBLqABt_(YQfgR*-p9ijZZ2sz_`-(6}rs)lYlW z*bt7y<&>Z&G_hppwqL)z`P(;LIbpMD*=>U*{wGs0x=Chc@X*yW%9nUl`JK37AQQ-0PDZe) zydZsV+^~WL*(oi98rmaPmMnsQ)4ZTUC7s{BFB==v&z<2b~0^R!|N(9hn*YPy z4bvXp^M1&*oXm>hneA2Oxj&1>4ci*%+XNQ?{|6Xvdi3 + I would almost be so bold as to say that most of these applications that we are building now would not have been possible without the cloud native patterns and the flexibility that Kubernetes enables. +--- + + +
+

CASE STUDY:
Sling TV: Marrying Kubernetes and AI to Enable Proper Web Scale + +

+ +
+ +
+ Company  Sling TV     Location  Englewood, Colorado     Industry  Streaming television +
+ +
+
+
+
+

Challenge

+ Launched by DISH Network in 2015, Sling TV experienced great customer growth from the beginning. After just a year, “we were going through some growing pains of some of the legacy systems and trying to find the right architecture to enable our future,” says Brad Linder, Sling TV’s Cloud Native & Big Data Evangelist. The company has particular challenges: “We take live TV and distribute it over the internet out to a user’s device that we do not control,” says Linder. “In a lot of ways, we are working in the Wild West: The internet is what it is going to be, and if a customer’s service does not work for whatever reason, they do not care why. They just want things to work. Those are the variables of the equation that we have to try to solve. We really have to try to enable optionality and good customer experience at web scale.” + +

+ +

Solution

+ Led by the belief that “the cloud native architectures and patterns really give us a lot of flexibility in meeting the needs of that sort of customer base,” Linder partnered with Rancher Labs to build Sling TV’s next-generation platform around Kubernetes. “We are going to need to enable a hybrid cloud strategy including multiple public clouds and an on-premise VMWare multi data center environment to meet the needs of the business at some point, so getting that sort of abstraction was a real goal,” he says. “That is one of the biggest reasons why we picked Kubernetes.” The team launched its first applications on Kubernetes in Sling TV’s two internal data centers. The push to enable AWS as a data center option is underway and should be available by the end of 2018. The team has added Prometheus for monitoring and Jaeger for tracing, to work alongside the company’s existing tool sets: Zenoss, New Relic and ELK. + +
+

+
+ +

Impact

+ “We are getting to the place where we can one-click deploy an entire data center – the compute, network, Kubernetes, logging, monitoring and all the apps,” says Linder. “We have really enabled a platform thinking based approach to allowing applications to consume common tools. A new application can be onboarded in about an hour using common tooling and CI/CD processes. The gains on that side have been huge. Before, it took at least a few days to get things sorted for a new application to deploy. That does not consider the training of our operations staff to manage this new application. It is two or three orders of magnitude of savings in time and cost, and operationally it has given us the opportunity to let a core team of talented operations engineers manage common infrastructure and tooling to make our applications available at web scale.” + + +
+ +
+
+
+
+ “I would almost be so bold as to say that most of these applications that we are building now would not have been possible without the cloud native patterns and the flexibility that Kubernetes enables.”

— Brad Linder, Cloud Native & Big Data Evangelist for Sling TV + +
+
+
+
+

The beauty of streaming television, like the service offered by Sling TV, is that you can watch it from any device you want, wherever you want.

Of course, from the provider side of things, that creates a particular set of challenges +“We take live TV and distribute it over the internet out to a user’s device that we do not control,” says Brad Linder, Sling TV’s Cloud Native & Big Data Evangelist. “In a lot of ways, we are working in the Wild West: The internet is what it is going to be, and if a customer’s service does not work for whatever reason, they do not care why. They just want things to work. Those are the variables of the equation that we have to try to solve. We really have to try to enable optionality and we have to do it at web scale.”

+Indeed, Sling TV experienced great customer growth from the beginning of its launch by DISH Network in 2015. After just a year, “we were going through some growing pains of some of the legacy systems and trying to find the right architecture to enable our future,” says Linder. Tasked with building a next-generation web scale platform for the “personalized customer experience,” Linder has spent the past year bringing Kubernetes to Sling TV.

+Led by the belief that “the cloud native architectures and patterns really give us a lot of flexibility in meeting the needs of our customers,” Linder partnered with Rancher Labs to build the platform around Kubernetes. “They have really helped us get our head around how to use Kubernetes,” he says. “We needed the flexibility to enable our use case versus just a simple orchestrater. Enabling our future in a way that did not give us vendor lock-in was also a key part of our strategy. I think that is part of the Rancher value proposition.” + + +
+
+
+
+ “We needed the flexibility to enable our use case versus just a simple orchestrater. Enabling our future in a way that did not give us vendor lock-in was also a key part of our strategy. I think that is part of the Rancher value proposition.”

— Brad Linder, Cloud Native & Big Data Evangelist for Sling TV +
+
+
+
+ +One big reason he chose Kubernetes was getting a level of abstraction that would enable the company to “enable a hybrid cloud strategy including multiple public clouds and an on-premise VMWare multi data center environment to meet the needs of the business,” he says. Another factor was how much the Kubernetes ecosystem has matured over the past couple of years. “We have spent a lot of time and energy around making logging, monitoring and alerting production ready to give us insights into applications’ well-being,” says Linder. The team has added Prometheus for monitoring and Jaeger for tracing, to work alongside the company’s existing tool sets: Zenoss, New Relic and ELK.

+With the emphasis on common tooling, “We are getting to the place where we can one-click deploy an entire data center – the compute, network, Kubernetes, logging, monitoring and all the apps,” says Linder. “We have really enabled a platform thinking based approach to allowing applications to consume common tools and services. A new application can be onboarded in about an hour using common tooling and CI/CD processes. The gains on that side have been huge. Before, it took at least a few days to get things sorted for a new application to deploy. That does not consider the training of our operations staff to manage this new application. It is two or three orders of magnitude of savings in time and cost, and operationally it has given us the opportunity to let a core team of talented operations engineers manage common infrastructure and tooling to make our applications available at web scale.”

+ +
+
+
+
+“We have to be able to react to changes and hiccups in the matrix. It is the foundation for our ability to deliver a high-quality service for our customers."

— Brad Linder, Cloud Native & Big Data Evangelist for Sling TV +
+
+ +
+ +
+ The team launched its first applications on Kubernetes in Sling TV’s two internal data centers in the early part of Q1 2018 and began to enable AWS as a data center option. The company plans to expand into other public clouds in the future. +The first application that went into production is a web socket-based back-end notification service. “It allows back-end changes to trigger messages to our clients in the field without the polling,” says Linder. “We are talking about very high volumes of messages with this application. Without something like Kubernetes to be able to scale up and down, as well as just support that overall workload, that is pretty hard to do. I would almost be so bold as to say that most of these applications that we are building now would not have been possible without the cloud native patterns and the flexibility that Kubernetes enables.”

+ Linder oversees three teams working together on building the next-generation platform: a platform engineering team; an enterprise middleware services team; and a big data and analytics team. “We have really tried to bring everything together to be able to have a client application interact with a cloud native middleware layer. That middleware layer must run on a platform, consume platform services and then have logs and events monitored by an artificial agent to keep things running smoothly,” says Linder. + + +
+ +
+
+ This undertaking is about “trying to marry Kubernetes with AI to enable web scale that just works".

— BRAD LINDER, CLOUD NATIVE & BIG DATA EVANGELIST FOR SLING TV +
+
+ +
+ Ultimately, this undertaking is about “trying to marry Kubernetes with AI to enable web scale that just works,” he adds. “We want the artificial agents and the big data platform using the actual logs and events coming out of the applications, Kubernetes, the infrastructure, backing services and changes to the environment to make decisions like, ‘Hey we need more capacity for this service so please add more nodes.’ From a platform perspective, if you are truly doing web scale stuff and you are not using AI and big data, in my opinion, you are going to implode under your own weight. It is not a question of if, it is when. If you are in a ‘millions of users’ sort of environment, that implosion is going to be catastrophic. We are on our way to this goal and have learned a lot along the way.”

+For Sling TV, moving to cloud native has been exactly what they needed. “We have to be able to react to changes and hiccups in the matrix,” says Linder. “It is the foundation for our ability to deliver a high-quality service for our customers. Building intelligent platforms, tools and clients in the field consuming those services has got to be part of all of this. In my eyes that is a big part of what cloud native is all about. It is taking these distributed, potentially unreliable entities and enabling a robust customer experience they expect.” + + + + + +
+ +
diff --git a/content/ko/case-studies/slingtv/slingtv_featured_logo.png b/content/ko/case-studies/slingtv/slingtv_featured_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..b52143ee8b6c650b4fe6c9d0b166b0f7163378ba GIT binary patch literal 8513 zcmbt)cQjnl+PCPviy9e3H)eE3i%t+VdN3HB8Acm5xeR^!EHUfq3$pYIUBeY44~!$wS(!xpf)JCUYHaH1{RltzLAHKrUt|s z?!piK%fs*Eg1ARxU`WaOAfVPxFb{wg%+A48n)RTul@;J%Bh6|kq6yYSD8TF;RDIoH zdcM!~t$m%WC2Uw_WdKq>kb40xFb^oe$Hmzd3GtC;{fAu0{rE2##0vNai-(gm>wg<% zq^S*1fV;y0BK)F2YXM;}Kum&PKol%2BESa_1Ph3Pz#<^9C=e_F0gFQf1pxoNSnsvD z+t@;M6_x*~>;6re)!xGc0Re%$y}kLph4|s_b|3)>2?-Eb5F{uFyypNSQLY|PAD}Ce z?OzfUVMuFt2ZVFYj5|>i~%ZZE0DS{;wL=+Si1pkFqb47YUU9DmN z!aDp9R`GwuLKNI#P!G7fJ{<1+uk2~t!#&_gdpH81pzx1u0k|}w)()D+&m+{u68S zAH4r3R`LIe1>GA1`kUVWBi;WZ-Ph3H;lGRb{^j504|Bb*dH4I`-ku(2#K3q|rlu&T z?=!O-K;Um?h&qti;74u@%FOVAj$7cz%3hHi!P@vz#|ykU?+h7!V5<4F+F_CcS*nG} zN6FJ_bA%ljHLa-^=2R;YFUUXNzU zO7^Hm!;N-Y^EOJ`0vf~ZhorKGLf-`Sz3phy$Ri2|NLAwN;^}Mzu~d=KbFUW5El)f5 z`QB#b2zRE|=@cewHoO1u4i1tvj&n53J0q(aZfbL+yxGXgO)6s@j($AV`RiNu4F21p zP(Pn9J^r<02s&QF-S2~1LnE}2JaZg*7eA;jhLiYE7&=r?`USZ}c?OUj?_HwO2jX?c z+Ab*TllpBf(#w6@Mwj@9rT~%>a;I-b-@`0_F zRvd{3uAcZVYrS4jO_%X~6O|a@I0*oDKW}-n==%;hJFI0A;+RHZP|xfd^zhu(+LIQ) zHghNtYQzBz$A-llDzXIj&H`3enbF5URow)pr=hW;@841?Pv>1E4_SW#uS(VwBb*=j zZB>aQeYE*1$$MGyv8NXB+;Wg_C+XTu9$gS+TOG9>Ac4Gq+@zeA{MTes?O3F_zA#fd zMdlJGp>0&a=QVr)cq}fMR9fZ`sMfQbx+KQ>>6Ml%y?7XC@VX{r@##)Wvv`nMxn?=y zGi$Yb$qBn_6tA)ffRR4b!7GNPtqjvqOJy&;Jas!r&pNR+U(t9dF1YNE8k=|exFIJ z!NRj4ubFjiU=+?J+lu|GPJzY2=jiK?5}S7doEfXuLNek;kx-%6F;7ygn=WjBl@SSW zd|qV}WL6||mmW)zowJtWC7_Neqjsj^V^S7sXDItVRwi#S8XZNPO5m{^5cA0dw@pLt zI(S8WI);p-$-z!%pN>nO+O+@o0M@}qTzw0hayG@MQoD}W##fdsL-Hz9>~}H0f1)JX zx3=l_PmG`PHMz58qV^CW=%(P$z4z7G`v)US4-b{SLVZ+JCf_7=2*ocw$(%FR$8^3u z3vpyaZ3X=h8@kpzd?Bk6nWgwuR>3w6Nf8o1%v8OpY!cK<9#1}n3jqaS2OLT~Op%Ck zRQh!T;VB$WD;w_q8R;)mA3u#6Z1+3|2c%`ESuW!<2)ube7qK2hxHcm}arOnjJ#Osi zU`fl0{_8g^?zSBL*`c^=I^YeHY2cM*SFdzC`VOO&3e*@ocoVjiYPrmn9X9-S2It@v zE{S?A7|)KhiiUBzoiT!z*Q2!$BDPXcS3Yh*CBe|vzU`=s?>2SF0M-se{i>n=+7iv* zK#%zYYgo||22ucd`S2K(-YYfA{uers5e0m+TK#bj_aSN?*E70+d;<#MgePHb4ctaY7@3%N@C@~ZRg z+{4=kCr=4IWhwzrK+#3Ktj8u7&3~N3^d9pLzf@B{$!+P-M(RgU9AE!N=aV1_{QRTa zYRckx)*|N?>|FM$%(nzgPM&*@Jb=6>T}v7k)^r_R3x}dy%$lnWV!5bLCw^mSQgL|M z$a^G&kd&MP+XC|7utho9c60T&i3?2MnCj%)?)wZRz}hbkuL))RRZzkR9}E4H3diTs zK>H1HouckSubpn6K2Xb)+Wf1;sbd*;Qaka>hNeO90ezA{gBL=&5rJ+^h%0o3%WCKdqw%2>b5bjDefAWAOX6PkP?mrzLO7ynN1#wKaeoQx#)1 zp=wI*=gf!Nk2{Jw$tAD9WM$HtS$yiV3phY5sF=D-CQY`GJ#o7;{qNEXckHz0km-1$r#?F5p6SA}KsuW=3}O!qqNEZd%VJ{0eOcAu(Oc z=r|ruhwdjC!iLko$DlKAj3)jV`z^BxT{F@~6kZkE^DX@__>;?V0!Njf>%y3-YeKaF0LwGTjE^!~b>JjXG%NxIqW=_4|v1 zqKw*@&&7TCECIeG8zXf>Lrgyxg+Uk7^}*VAPY|&*ZRUyiJXE5><5oFJBZlWT%mI=_ z3co)nJ6Wpd2+(YqE=P;XvxL9XkV=_@Kl*Xo z(I%{lVOBme7reWzhQPhO)W*$tUj`x)bF^mOQZU{?nqJkO2SeNPsyWzS0@yMtrkk%u zm!$j~B?QN%&xBjC3C-J?CuM^NZC(hIXk;JMtv{B@cwts0NlN$t%aSjicI&L{OeTOh zH?5b#|NE<86rLW=^F=JYs`GvIV2ZarTo2wFX?M#y|Ge32(de_Xro5`nKTDwX^FpTr znk!~JCN<^3P|?gEi)dTHHORi(W4AdGq9l-BRejH&nf)gh^B=ddvjolOZgoW~U7uR& zlm-EhW|P+QEprrfzV?aS6_T!$&i9sw^nC5ICcLa;n+|g7jSs}#SDl$`nwoclY#WRJ zIvjYV7&vBpJrRgsLQ$|^NiQ{RWaKg2%G_(~vLE+ukwv@RXDNl-63!;2vHE`vO)?BkXg=`=lW>a89CruQ({ z2~o-2i-r^^EkojnH<_)dCp8<-h>iYUEhUv~!~GPi%IrFBVfuca(eXX+PbXhfgbj~}Jj{72VV6ssBD}osS)3a@ zS*NZw)q_XX4i}4tbfrC0^~;^r6XxM^KQf#5O8VNd(__<_YHjl06qfV`3p+Z4=ii+r zMjd}I+HA^ZqR-_NIShs;RWoCu;*JjcuRq9ediy1H7+SaU) z?@~({JUNB0{M9nyZ{km>GgIQ90BZ|+Aa6z67I19(c|P-s52{NmiLV!ZmI~>A4F7qz z?ZcukVINM!t7`*sc2x}w3We(!5pdJB;CDPFmvI>W&=B3aCuq48;(BNXIZLRXPpnYC z%5%Y7&eOf$7lQ;*r^9ISe^bei0u1p4elG z92iJ=rh(Dyw|GlTXQexR5ppudq<3IOX-=1zOGxlT!5;@d_;y2h&!_G3h00M|DE+!^ zetpZ6(gX8=98EA0bm83C*+{jkr$f$A^Js`rm8~aY>}EB1{ReecHPAt(NmO({Y+ACI zRK1&sV>Mi)rPYEa^NZgp$=DNWgM9ji)A-`tV(ab6^ZX}a(xHTa&S*}O;YKaw|N#){MJb^oqe%Sqc*1pNZ%TuNR4IW_WQV)OB&_qFrX0MG6i^elBNk>?3 z7H03oKv?~kDtqmSL8n(8pD54+Gzm*aiC)9qSg+5HQL zWleQr^rsay9gFt+mpB_LUoW$mff_`rL?Oq=Q4Op-Wl=n_X-~9{SqykmLBjX-PJMPuUL{5DW9ck5^6n@!<657`k8hY^t#<`;Y4iZqwrv%OJF zeL2Xq`^G6n88fsI1mjMuduBjjOldGEI5e7_;3=}r_`2vIO6Z6$jIp(}w{=T#x%D}N zFKo@Lp*pVK+)I5TP|_MMV|=}^yjvM{oGKB_yAu%HTgbRP|XvrxV zO&UbFu}oe0O}_A|pz3ko7Y1%=>ym9%eRj(D9~b*H3G5W?2N~0!HbGMB))=nh;Nx-T zm9a8TnQQlhlb5z$x?KYOUzs1>`GyNDI3-j*^D`fJqEw&)K!(}!ievrHE6B_F`0jqr zOs$HW!XKuxKWfc{zsqwtelbAbuZUMl(irfcT! z{1Ol!6r~tw?{ZqK&ISgJi2{?0;>?AYe!VI_wO8M7F!XICu7JhTj?>xFYy%Lj9mw|x zysz7G?w-gM5}8V9;9*Wp#Vl>aM&xSKfV(K~UWPaWK`>p1;KH2CT$2d5kCCge4+#l6 zE+LvU{jJLPhoQ6|ZA3&q>Y+zXJYSk!{^AQT%{ucpJgSeAAN{SNZ>I7#qxTVJ(A^~G zv8?^LYsyzDzD+T}R8$`kwj1hI$Wd>uwKx^xeVE)OTNF=Y#Kz|c`s*g+2Om`%bLDQ1 ze`}jApjxUf8cToYlwk3TTKIi~EkXf~I%iAe$=x}@1^hD_@T$`9xZG$+!k z((~*@THfWX;N12X&LHp13`bRHC|?X~yfmTJ6XnAX>8LB{x$b)1j+Uvi6e57iOGRyd zK-9@uXr%~wwrWRy`W$gYoT;yyBx)U5DXIvzOWJ)l?d-HzZ*+7LM|1d~;x~^mQ({@x z*Sm}M3*?8%`3iBfyd%d8!ZyM`o{G4LV(wC^6@4+{VBQ2Rxxx|AI(Uz@Sk5|wuVhiIwwi?S9T>t-(=X zUz4bhGBjLjR>!9jC>zH#S%GLA#S#D9+O zT-2oyV97pPwnkm_r^U@HO%;|R%I?8=k2EmTfMiyX5b6qTV>m+kO})gE{_Y#ld!G08 zK=XvXLFOyr))Rx$A&-bNm3Dqf+A!+YC&$7a^J`>cjyt~{2W`J$QruA1+gbc+0KN{A zH5X_S;=&?A*EFZb!?yxzX7RZ;ge_Cb{g>~zefNt}1Mzmpg-0akZv%^iXC{vq=a9zD z2P>L?ltdh#u1h!``l=h_f@=!wd>i%9b!0Z?n>-QC4@KIYTZ`mqiprVBu2+9XcT6^q zVo9B579zyFM=PbVTZZIZ>l_>;0fR2J)bXbM%*JvP%tbMOThV#%fVM z3}J8@kkAPyc0vWu&$xkmTle@{h8=!g5ky%&*M7hs(o)UMtNY7EyfJBlv_&=klKtQX zwdJFUNhg_vDs+1?NsDYcjZYEqap~K+4%a{m+(IC{uG&ZzLDTugD=p@nI05C|8_X;= z6%fMo7PXO|ug0mFm3AlcQnE?-I5yx5`2po>!ZRSvEt-0(>XA|;&DB*tg5A(@hPCmA zQyN)LA8jr)-pa>{#xEl0qdJ{b?})5Kqr)UB$J%?vJUEI=*BA=LpQ=1`M#s{Ote`d; zBP#_{pS&nVWP|vKud|GU$8f@vujx_{4V~(8>d9ZT@0JY<I+!@6(50b5*PiI{1pf!Kstyhv69mtUQ7CFRyD+!9r-FFCMt) z$nz8Eo|dz&b|i z2?QGPqM!Qp(`>wkDv839vXhrac@v*)-i^MRLKu#s&6F^X!}Bypn#6`A5YRyi%U1tg=HGs9ke^JEAXip!g($~f(EttwWPt|hfBF;C(zel*0v)k8G6 zL$K53`jHenXe9fc->&PVUwByur=I85w({N^-$jkgcbx4lm6A7y2_<&BMmubhWv(g0 zd`vMh1HL4LwhhIb-+>DFxOW7--zGTQhEjZQ;qvI`54fga;+%wW9T#KoniO{Y^A|WD zq$&f>yEDJ^n7F$q8~c z;Y%3vmSY9`l$d|HnZQ4e-G~bua;4ZvVvwUFP<%wuCLdgsCw)-80n|#yn{yt1@|VD7 zmp4)3O(u|@Bk{nvTswPvDy#NYlx{ehjHu)Ofcu3v;q;iwNfTonEe=69Z|)n*oTr{g zCLssT2Op%K(QCZ1#akoTO=2o>(dCfpg>T6HCUyynobo-bU&oiEhl`yaVP1!LLZUxx z%(Y8T=`Wo6tZHtCSf2rzJFG!buhMS<>bA-&l4-0KOb9Z$4psW`MkiXTGb`-NLvpLQ zPSjRUh~%jn6_z`z!0@V##y5{}2u~au__a=wWL-}!$R7uew@T^wobFwZMvcDkxV1(3 z$W5PU`rO^tRi2lYe4{t+%U_(!a_F6oO~dYN@I&P{VXkkX+xtGdJA5_Cow>oQvM{6L zvs=z=)C+2q(mi0Gv5TX%*cY`4ks00$f@hkPO0}FAphNX)qMh_XzBxgVj2!{zkPD7y zb%P{6uJR-Aymer?E z2MZE#+6c@EC64%FidJzAL$Xlw3+3Q%;;YhM6CXCZnIBq@)jaD7CV;rZpIq>2OoM>L zFU^V#{`Aw3%#7__QWDJ`Q}3!OqO(o9tv(K|OGck^_8e=Ok@wKNX9ssK2}&8>bZoSG zKf%MQg$81r4%|meM zK}KraJT*0LJn_mZlZLDaz+1GpMizF86NOJ(Ir-o86qW1P(pax1oPSIJ2Nuj;%Q@ZN ze@mX`xiLMy<~lqH3@kY`QomJkK@9~kFH^o(dN+ zoF_z7NRDTB>>D%Vez;p^3m0%GWt=0JI1lP}`LkSvv%Oov zJldcB;LEbV!!;E&nr|__#Hh@cDj7HN!Mi|`2z~sSkOw(cIJs2)XkglVPk?2;oNM;J zdbzp5;ht&LBG7qAnboh}^Uuc(zs!?l#tra!eYE;aEH+FHY8Rayu7rnP3N8O(VQxKm zYAdHos*1gNkviS5L71X@*OoD$epvQd?=hr*ribR^u-;C7cZZ~RmNY+hAbH9HVN0dl zRcHVA?G!>`|87VUqg>A|*JV*uc9DpLJG@3guv}?EI-zd}ubURLgnfy(4|PS| z)xN@woN*fvrdcR1C-o*~x`1q*;%YHJ!vwUkhtB8U?`B7xO5gCTT}p}XmcF~O&8mPD zmB#R_6;V&Min%CE=@`oSJ&3p17G1Xf@avL6EG$j;5_oJ=+4-=7-Pl%6Y=%3Fr~7t)XGDgXcg literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/soundcloud/index.html b/content/ko/case-studies/soundcloud/index.html new file mode 100644 index 0000000000..50611ffd85 --- /dev/null +++ b/content/ko/case-studies/soundcloud/index.html @@ -0,0 +1,4 @@ +--- +title: Soundcloud +content_url: https://www.youtube.com/watch?v=5378N5iLb2Q +--- \ No newline at end of file diff --git a/content/ko/case-studies/soundcloud/soundcloud_logo.png b/content/ko/case-studies/soundcloud/soundcloud_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..f8c12f05b5edf6aaac4ef457ca3eae367f2ebb98 GIT binary patch literal 22800 zcmeI4cRZWz+y7&ae_NJ&+qeN_?W^8J;RTMQ!6}8nKrL{-xJ!{k2 zd&erjXz!8x{@(ZFd7jtv`%hkx9M0?WIgayspCcD}CC`-=rHJsU@Bsh-k&Lvs3hH?f z_2Y?)iF({pt~NkD;n_%Q*#Q6qq^Ca^fS9)w001AGnVP1(rh>cx6mG?4U<5aWu_3H% zP-*}GER3)*fLg%p=?q~eX4XRVYh_jRbY@0E^cvg>>8ZG_fukTAw+-wv>=_Pf-;>L+!jX1 z&BntD<=_O-@$$2AaPx!sxLD{w>>NBmb}k?XKPv~P0P4=gPWRi59)gdGz_v!l0xIH? zzm-G%CPZ&)Z*L<21Ufl6u{m+F!EH@Yc7A>!I|v8@v7!{LcFxxJ1_)MbJBD9Le$OKg zvxC~2+1Q)Gt?5ql8W_SI?1kv*PYeBV`|Z4}Z2l-@ZTA~H6h$Dyzy`>{#t!@^lM(a} zkBx(^<|boB zM}FCUPK9411V){!fS4`Jz#eX^28UZhe$6uF-x1P@iJi_WZZ>W@CIthine{2zyI|no zZGVpWpBZ7|2KF$>>6~O`=SSVuICum=d;%a&R(4(ic6PKRzm$K=p#V2BGj{%m99--I z+#KjR(3F4Afm%aG2KEO3tt{wTv^+*o0b{tWm4Q9P%*wz72DGs@0R#VRL{t7TvjoK8 zmT+5Cw_p%XFz}z6(Of@gn}Ce9oxOoI6ec4MK{c?MnHdRia`PMW@Pc?)Ie0m^S-C*$ zoUD9C#+@U`u8MuaASKX16$aA6V#me@2>Y>ng442dkHY`^i{yp?5~6ItM#vA^!u#+&oKp` zZf*jo`GD%1!OwBv00aM{?a#`;^qQyRYU%J(&kJHl-GCgYkH1;b5C7e&^EjL&hMi3nYAjtiZ}nejQUGZCQ+&T*mBI5R%SbtWQo!8tB;8fV7m zxXwg`E;z@9PUFn@9M_qM&;{qX&}p0*pW`|c5xU?U7dnkI<8xeRB0?9O<3gu#W_*t8 zOho8{b6n^&&Wz7-orwrtaE=R|#+mUst}_v#3(j$&(>OCe$8{zmbip|;bQ)*I=eW*9 zgf2M8g-+wl_#D@nh|mS+xX@{w8K2`i6A`-L92Yu`GvjkyXCguuoZ~{Lab|pu>r6!G zf^%HxG|r6Aah-_>U2u*IoyM8*Ij%Dip$pD&q0=}sKF4(?B6PtyE_51a#{VTQ{69Yg z2(w0g^3MtN5x+Ki0Z-J&1L>gBDhdFA%WVL_%Lf41KS4b&000i`0Knp706-ub03e5l z>3@+#Np)nz@2eq(m*bqvVHywTs;b;qw`Y-y56z`hKzP14v9PhQmv23zpcKG=LB}DV z&H6R?S%Moj70w!wY-r&3GiJ$BK8;TeYN+^sZFa-y%apU3NUD;DKTpyp z3OI>~h;?&_N0P`qrdwomc! zvLKT8$8_V5fEhz>HgILDQzK*PY*nT!wo}zc=R>Rr9wlO)O(!@j_O0bz&rK42svM7) zV7Fq_8K1}rhvn``y=$wglGAq~;!BK}pHaWE5na{2TjMXqRZ3EMLUmFNttHnFnn6Kx zW3;5+{EG8c(vKBm-o3@CYyG6t3 zzunG|MuZSjhZKTzZJy8fC^C}3eG%7Fw>xajze>d&Fk!ej-B`~oyU}9fY6^nx>vJ!E zSR7Z$+)s^f0vKK&2iO=Ne;k0T4j z90Sv5uYnM0WjID3<1>|8T&Z4`vs|CqKV45KlyAeEUnc@&8ZglZnk%|&HVt977;U;#Ma2WX`d=-_EB8Rk2tu&FRP_HYrMVN z9WmV&g3M40c>NHwn1h--29RH!P!-CeW5cvwj?+~s_Ou-9@jJ&1eu6yP1ZukC)cKs` zCy7XPN+RR9`VN&l)J25Po1~R(Lylx8zj_H&XwX(3MZWN3EIKl*Bhs%Jxo?cMC}LVo zw{HQ%#A?K6$Dx=IZAoWc!)ekMZ`#mI=hsaY@uzQ<*1N@TL3>BqkY7_iu6J)u57fG$ zxz8TQT9r2n} zA1)2(Dt>GatjwzBE;51n3bK3OoxRc19nM9{n?kvtTKe7RgesH&^8EucRM$nw?l_I= zUc5@MR`k47bco6A3CSdM>m7qs+_mb%>wGQRrK%qu0Mh3%vw-|fcsu!BR%yz-^*N1@ z)CVb&Nq94f^z)5ZIrf@Pphc_Dt!`UD*_2SrAT!xJFBX>1)32N`ZfUJvFfVFZ(-Wx* z2Nz&J-MvXCzfAY-LqNVEpJ`z^0m9_=^Sk2{Hm$0)b=++I?9SBB!54S42=j5+5Yd7g zpe*;wPc#A7_QZSO;Iz7FuV*{^Q_wnPA4g>PWtkLs)<(&pQbR5 zkE$i=ZqsE#QsdY({PhSYzs{44iH!@{)Y9VsE7%Q&ESXl(sgqBczVmJhoS_ z(6p|&$KwUShevstG}Lp0yvYF-*REZDyiZKGJ#3rc{YHIt^D6DjcGT*fLAYM-g%DZJj84<%V&-ZraeMPJvY$OSZj7#D zu-;&rd)Tl1qN8l-^A7^I`KWFGtvd3F2lwKx5mXUeA@tlKWL;&}_dMv_YdLJmEkd{= z*6eUViRB?WUNOCTrQ}`Fn3fB~zMm~<4hM9@6TD_`iKGQFM9m3BuCevkTaL2!JbI9s zO8kC45>H@pwTa$eM{E~?-J4fx^ID}&*ET5ZiYlcAiPW2K4h38itZME6p+Cyg3=B_suvxalQSn3a{dubeorwJQR- zYB`^=If?Sue(&N6$ zZVSA$y4gr{AcWh;*y3dj4c*10n|6*W4+sR!M(D_7T&6m14CSAti2h#d%Ro9ST7w;C z##t4h;+fc^&k&;L$gJu{^;-PdeCg|&INk$VeZ>3HS<>fzi6yfHuCU9mb}FtHyYStX z5H)FzTHTJuswvIu5@EFR7e|~AYR^MO-ah_Oq6aMw$+(54 zCN1P@)`MVPer31jbD9jJ$IbYSnbsa`7gy4)d*pQ7lS29t()Tq!Nn>hpUS3Js=1ul8 zAGhkY_`tM&iEwQvLkSV8`A}oPH^C@Cev^)%6f7ItMw)66@2%T5S^#3pXW3y4-sW(A z4Nk6DP|}lg4;FHk*3V=y5G zGGDSCj zZ8!^`^L<3P#`{ohLT8GOoZ;6FJ+F8bZk0|gqJ}o$%ZS!z!7LzZ8%TdfymmoTzd`ws zNkTtgSzMCopfq6IpIX3*PB~nRWj;5JLzV8;+_vEO#H&H0n!-y1WEVgM!xZ`J<1QH( z+0?L|bl-9^7b~9OX+JRTJd1_Z<)_rVN#nAziH6la;K}>`ESHE##79Ll=ke6T14M{& zr??G?a-Y*NA`~R^=tycT3aIpY=O^;MJk7d8qSqmU>s@expvK}Z(T#%YL18`EdT=M% zxMSRm>4xI0WF@TgA6uK;j18ZIK)FfMB^gS}lh^`F<7UicDZBX}v#iTlCPHSNgRjYY+Xw@fU~S;q zVGx^EH4A@>nnm_B8IhJW4zZgBKFo&sBL5X;W>au7aV&(_UGc-1-BUApDM~W61HI%-+2QldxSowNHteeV+Rl0cT_nlS=2PZUoMLjd?dChJ zQ=XnaLAR+H>D;U?C=GHdSW;zqOy93_#@5qjMgpQ+ZHQf>`Q4I4<_--e&lGWe+Zt0<%{v~crm}3s zG%|5#@IH*cb#W`*#pjaI<|R6W#nY@A2}S!wXkQFPsj_a+U2~!r>9}@`q=%@>e=UnZ zs_*ZRDhaX{!|mb6jOz<4n=hjaOmo@^6R2O8PTnP-TS&11b2Oy9y5h{n2&&3N)tKXPyf z#?UZahAZ5&Xb~r5LO|tr%z$8oWVS^1?M_jAsXKGeLgzPM-4Y^Xsq<;eCv0tL`n;V? z$x&B$OV!mrCAy&iv>fy%3w%A&ptkUq{YYxT{HBO+dZ<&gQ6gbO(&e!^M-3J*B*g7t zpJI3eAi^m8n%dYA6XNuOS|i+rxoz2oWVL2VfeOl1a;f=aTrf3PnC{ zp`h7IF)V^~<6tFgneGpr%SHK=+$OLZFv6V>v(M%#l^*;7t=xlWO4j{-N7Y~6Ym;&^ zmJW%uwYF&8g$2D#QOi}SEqS|S#wv@i}Br0UbQ zR;IKz=>0S{VD@SdS?`D@MA}`Y9ZwR7M&F}h~l6M(_J(0xH{BYMJClAfC#33J9T zuh{5B-f9ppu`OyzuY^v!$2q-ex6B1^ZVc7t%XE0WFAzu=@Wn3cz>FTdPwNQ5p57tj zqBC_PThG^`mv6(|KEQ>`t6G@bs?%JRi|?2&m(s8rHCISgRJo@%^!a)Mf6&eT^6tLr zx8%g$uQCSK^C?G@FSc9-iPp6@HKcRj1tS)Ocnj~CxfvVq-5zRJ`B>NHA4eK+(d={R zkdyCpsH)NXM5cLIi_vyFm*AhL?I&RoX6SDc^g}o#GJ4osc0pTVP2ONLpS<^KO5!bjdqH#qOw?l!FH$ z(zDz9W?0gKx~or}P|@R(NQL4+Pd!;fnm2o)kR;cq3y%#YiG%RG^6zq|Rt(BqRKlNS zFUD#f+xSebYc+&-_kyMT+hlE3Xm|lY5qhPo{;gpB%Y3}sH#n$vWZ*AN)HUV7`1E{! z>HNeGQrg1qB?oAQvxi}Nvonn%v)bISNAa6y#N6)ye5vv&6eDG?Q;*b?S#=zD?}YQsdKje<eh>3_SiAC6(}aHquFHO#o{$hu_GZUkdi=}ye7 z;>D{V?OuF`a6tU<)0*kX`US4*szLd<@@@Hlrig{uD zZ9=5?sLB_DNz9Oxuw}NE=d2P`JJ0VE`x@r?`Vt`P8I{s%!58T8w6vt-Y8jkl^g2Xe zCd&ZGSSPw9I-dhw7|h&M^4B5826m&KdqXpUl5&M*EjB8f&7VNjs42*H!Qy4a9nL<} zmsm#J@emn_)%GXVuDTeiTY#XAOZgr-%|p{f6(V+ma#Oykjqes&QDFsbzb}iJw}Key7``H`ltpznj_;#HnlS-S!HM9>!#^co<~X zQC7O_*dkWCAIQ~+L9;GR?=g@BfPdthSmH~mbbZ~bjDHKqVQY?m zE&w37hWDa_lAN78Bw6Yt+4G>{7%Hrk-Jz=2G~%I7A6J>gH$341V9(A~piIA!PBW=>~hPYm(u&PrP32fi$}ypOfb zCXtMzPT_GygMO ze9@>opjcPTxGc#_yrOxTXVz)&>n%o>Op~YT>1s>A{*V1W{CnNz19h_Kx3J?!o5>Q1 zUwrDDmSrG3Ani3|-x9*&tqb(4Xs(#~6Acq0r}*e`!Zz2qS4ekTD)G?XF$!Fc%?+7V zhy0J_xqfB&O-ID=;2T?#P}08DnFpiVj|?2Tud_}R=rYj9^9+|B?)4cb7j{Q}Qj)*v z92&6gtYY=8k8obRJe)C=RY$k!cwM8m^TM z*FWou40`go)-V*##nZBRnwEYWT~1LVi9aZFZQCW{_24)<2zMSA+j(kN2h_V#x!nv( zty+wm;-B;F^jvF!tLWf4s4qvBP4!o-uCz*yigN0UmO3ei^j8`v(6!&ZEZ<=WJIce#rmD4Rl7z= z*V603_Z7<@q!2^B?$xaNkGA^#R4f;eZacWv0y_o}?lt`ie+>wyE0UH7?#5RduV3aH z3yF<8OmdlqRjyeaUsvj3<;Aqj+HVOhayjI9(c8ba!~5e!iAG_d$DOw>m7d44726Bj zWg<2v`_EX%Rv1ERxB4Z917*~=%NrLBz))me?bq_kJpn`L8Fp$|XjF0^ZwCTi6YU84y=1DI%-9X%y|?Cv<$6>MEUI#o*i(uc zdP9=hv?JK*NaE5i^+M_gJN}fvKEA8<=DwAp_GiunsK>6h?l&Ufbeq)XFp^x*v3Cu& z<`VZ~XU#;TiV?Ch2k2AI=FjAN?@!W^^y+dF%}LfbzueT4%hrdVkLRO)gy>(T`5``% zUsm9UWN>qd%&JLrFJ_(~+Z~sOwz2I$iI2mXx3(E8UUo^>b<>hPY`}@TPGd*uGJ@-1`6E>oByS8>J%m_P#LFZYdxgriww_QeWXV6@6zF%1CfwIM9ar@nB zbx69Q%@0t2y|)nH5FUyX=QvP8D>oYMcbuZc9Nuo7gRxxvS=(;;D(@|F7(WF)R3CYf|{+E|76*@xD z?dy_u3`X2^rACr2m|CB6dmqh{FW@OhntrY{p95SklvDXMHYDc-o0C)B#qd2sHMve^1uWc#v}gG&k@;$zg0)-V zMiH$qwQDc*$K8!L-fMh~6>f9Q>Ag#WRH|g8{#>U7QBWQES~a*8n$~mUYm!%75$WSW z%F6E6O*cEZoxES6gR3(up&BFEMvuw-Te+i(+`?S97BkB$@X~sF`u4Gs4Y$VATl&CS8s~E5i05g8r_}Y5Z0q=wCDQ-fH zr{vqT$*mOr3Z&p2(F&Yu|pc zVfk@^%T4o0GvHDZ?fQi0&7C$j576Wz#U4uXgDN+^qq>P#$b5?9sUK6kp6dElImuZU ziZzM`j2+%YNEjB`(WI#NP8{AD+a($;9w&yo-WYq^sPFGv?|i!XX;cQjuNErXH6Pi! zrYJnp%kZehB!%J!gjsHi_PAl<_+D{C>HJ+r&3fnBnb6|TS#nO3IlZ*ayQ=&)ffEi% zY%c08K@nN5=clTZkZ8zrDe$|c+?@!9 zp&1~v$4U?Le0K;B4P=YMV=2u{(R3-J(w=vP z!N{9e9~x4ccVu~J{e9E!Dg%kj+nDdiJ4)}XBY1~8#)cm0Hwt$o84b}LRei{bwwhQ(lf?1I4;ABp+TqN1`%L;@KLqHAC(ZEcE#bQ$qgGzx zU`&zHb~DGXWZ!LH*-Q~M1NKmTZy)^niTtf5lFRiAv)+@ZL$$_%a_?v&%RDC4iAS?$ z5r}UyyJ3S}R9NY}d)+4U!9K&Dg*Pf0z>^&12P=-_Nm6eoB@Em??5=G*s?209t1I*< z-t&u!Sa@MN)!<3MIdUW;ERlV2Ps_w4Akru1xX(o8Lt)XS-DRjtnP}H&hfTNASLTUX z5{T1QVY_uRo7;<3aNEj+(SBaXr)RuB;#LHRx@rXZ?0tvA>-x%5XSqN51;$U0I3Bzq zY4MnJTN$gf(r2*nO)ICXJR*{ReE~FXx`?M_a-zGuFAx}DKf8ZaQ?`;-DAxl7+AfW= zm-QA>*_9y{w-Eg!%Q!S{-!kA26%_o;b^iapdHyf&yjNl!BLPl0!(Z@0lP#y;Su7)= LD4rww`04)xu@gFQ literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/squarespace/index.html b/content/ko/case-studies/squarespace/index.html new file mode 100644 index 0000000000..58c5cae5cc --- /dev/null +++ b/content/ko/case-studies/squarespace/index.html @@ -0,0 +1,101 @@ +--- +title: Squarespace Case Study +case_study_styles: true +cid: caseStudies +css: /css/style_case_studies.css +--- + +
+

CASE STUDY:
Squarespace: Gaining Productivity and Resilience with Kubernetes +

+ +
+ +
+ Company  Squarespace     Location  New York, N.Y.     Industry  Software as a Service, Website-Building Platform +
+ +
+
+
+
+

Challenge

+ Moving from a monolith to microservices in 2014 "solved a problem on the development side, but it pushed that problem to the infrastructure team," says Kevin Lynch, Staff Engineer on the Site Reliability team at Squarespace. "The infrastructure deployment process on our 5,000 VM hosts was slowing everyone down." +
+

Solution

+The team experimented with container orchestration platforms, and found that Kubernetes "answered all the questions that we had," says Lynch. The company began running Kubernetes in its data centers in 2016. + +
+ +
+ +

Impact

+Since Squarespace moved to Kubernetes, in conjunction with modernizing its networking stack, deployment time has been reduced by almost 85%. Before, their VM deployment would take half an hour; now, says Lynch, "someone can generate a templated application, deploy it within five minutes, and have actual instances containerized, running in our staging environment at that point." Because of that, "productivity time is the big cost saver," he adds. "When we started the Kubernetes project, we had probably a dozen microservices. Today there are twice that in the pipeline being actively worked on." Resilience has also been improved with Kubernetes: "If a node goes down, it’s rescheduled immediately and there’s no performance impact." + +
+ +
+
+
+
+ +

"Once you prove that Kubernetes solves one problem, everyone immediately starts solving other problems without you even having to evangelize it." +

— Kevin Lynch, Staff Engineer on the Site Reliability team at Squarespace +
+
+
+
+

Since it was started in a dorm room in 2003, Squarespace has made it simple for millions of people to create their own websites.

Behind the scenes, though, the company’s monolithic Java application was making things not so simple for its developers to keep improving the platform. So in 2014, the company decided to "go down the microservices path," says Kevin Lynch, staff engineer on Squarespace’s Site Reliability team. "But we were always deploying our applications in vCenter VMware VMs [in our own data centers]. Microservices solved a problem on the development side, but it pushed that problem to the Infrastructure team. The infrastructure deployment process on our 5,000 VM hosts was slowing everyone down."

+ After experimenting with another container orchestration platform and "breaking it in very painful ways," Lynch says, the team began experimenting with Kubernetes in mid-2016 and found that it "answered all the questions that we had." Deploying it in the data center rather than the public cloud was their biggest challenge, and at the time, not a lot of other companies were doing that. "We had to figure out how to deploy this in our infrastructure for ourselves, and we had to integrate it with our other applications," says Lynch.

+ At the same time, Squarespace’s Network Engineering team was modernizing its networking stack, switching from a traditional layer-two network to a layer-three spine-and-leaf network. "It mapped beautifully with what we wanted to do with Kubernetes," says Lynch. "It gives us the ability to have our servers communicate directly with the top-of-rack switches. We use Calico for CNI networking for Kubernetes, so we can announce all these individual Kubernetes pod IP addresses and have them integrate seamlessly with our other services that are still provisioned in the VMs." + +
+
+
+
+ After experimenting with another container orchestration platform and "breaking it in very painful ways," Lynch says, the team began experimenting with Kubernetes in mid-2016 and found that it "answered all the questions that we had." + +
+
+
+
+ Within a couple months, they had a stable cluster for their internal use, and began rolling out Kubernetes for production. They also added Zipkin and CNCF projects Prometheus and fluentd to their cloud native stack. "We switched to Kubernetes, a new world, and we revamped all our other tooling as well," says Lynch. "It allowed us to streamline our process, so we can now easily create an entire microservice project from templates, generate the code and deployment pipeline for that, generate the Docker file, and then immediately just ship a workable, deployable project to Kubernetes." Deployments across Dev/QA/Stage/Prod were also "simplified drastically," Lynch adds. "Now there is little configuration variation." +

+ And the whole process takes only five minutes, an almost 85% reduction in time compared to their VM deployment. "From end to end that probably took half an hour, and that’s not accounting for the fact that an infrastructure engineer would be responsible for doing that, so there’s some business delay in there as well." +

+ With faster deployments, "productivity time is the big cost saver," says Lynch. "We had a team that was implementing a new file storage service, and they just started integrating that with our storage back end without our involvement"—which wouldn’t have been possible before Kubernetes. He adds: "When we started the Kubernetes project, we had probably a dozen microservices. Today there are twice that in the pipeline being actively worked on." + + +
+
+
+
+ "We switched to Kubernetes, a new world....It allowed us to streamline our process, so we can now easily create an entire microservice project from templates," Lynch says. And the whole process takes only five minutes, an almost 85% reduction in time compared to their VM deployment. +
+
+ +
+
+ There’s also been a positive impact on the application’s resilience. "When we’re deploying VMs, we have to build tooling to ensure that a service is spread across racks appropriately and can withstand failure," he says. "Kubernetes just does it. If a node goes down, it’s rescheduled immediately and there’s no performance impact." +

+ Another big benefit is autoscaling. "It wasn’t really possible with the way we’ve been using VMware," says Lynch, "but now we can just add the appropriate autoscaling features via Kubernetes directly, and boom, it’s scaling up as demand increases. And it worked out of the box." +

+ For others starting out with Kubernetes, Lynch says his best advice is to "fail fast": "Once you’ve planned things out, just execute. Kubernetes has been really great for trying something out quickly and seeing if it works or not." + + +
+ +
+
+ "When we’re deploying VMs, we have to build tooling to ensure that a service is spread across racks appropriately and can withstand failure," he says. "Kubernetes just does it. If a node goes down, it’s rescheduled immediately and there’s no performance impact." +
+
+ +
+ Lynch and his team are planning to open source some of the tools they’ve developed to extend Kubernetes and use it as an API itself. The first tool injects dependent applications as containers in a pod. "When you ship an application, usually it comes along with a whole bunch of dependent applications that need to be shipped with that, for example, fluentd for logging," he explains. With this tool, the developer doesn’t need to worry about the configurations. +

+ Going forward, all new services at Squarespace are going into Kubernetes, and the end goal is to convert everything it can. About a quarter of existing services have been migrated. "Our monolithic application is going to be the last one, just because it’s so big and complex," says Lynch. "But now I’m seeing other services get moved over, like the file storage service. Someone just did it and it worked—painlessly. So I believe if we tackle it, it’s probably going to be a lot easier than we fear. Maybe I should just take my own advice and fail fast!" + +
+ +
diff --git a/content/ko/case-studies/squarespace/squarespace_featured_logo.png b/content/ko/case-studies/squarespace/squarespace_featured_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..551b6da32119dd8701405202ab8c861d63966a97 GIT binary patch literal 4539 zcmb_gc{tSV*C!MzYY|FgYzZ@F%rFLH8_S5vlBMu4n@pBrX6#9cB1=h;JzMEXNF=gk zP?oYpDWvQzWXm?T-`Dfh^S*!d`@QdbUGH4i%=g^qd$#-B=X1_|O^g-ZbT7XoKL-cL zUYr@$26(;$zTvz(fZrWX=~KW%f@$o?w57Q-0||5rhXI*Jq=0bVgwqro3V|H#*Gkdh z;NYTDPdG9iEiKR_nm2^7EdvSi_64*#ICS)ad(wx1Kt!S0Tkrz<-4N{Viw5XzHWUo{ zON8mE3;sJOM@uV^5sgj(At4&7Bp5;sq^SjgX{aHPFclD74WOwR|Orl6g~A|e zP;c+;ef=fPVA@drD~#VtGfo8iQlK^z2F;&N0?xx-_Lnk{yZ`KHTM&o_eVk4O4u#-_ zrIGx-DLzabRu>GsfsmQ5x^~=uh z^sz7l15Kocmb$T_F$`vm(n1-)4N;m%ga#Y|L%@xG>Ee7COo9)I^2;|B@cmO4^&fT7 zMsx~+Nu!^j(Y$_Dft3f1Nn?1>d_hJ=+oz6%AVEhg2_&k|Hd|@?sQ%tLmO`fnP{<~9 znm6dLT%)PKi9=lrg+;;*wbbDz$n8C#j5Rfk4Al*g>PA3p1}5O2y5!&F@29T8ztx2T z!9ceQ<-ZlqFBVXZ+vGnm02coMiQ)sGhYrBRYB_8efbK^e*5E`?|5QfcwFC#L=Zi<~ zV=K)gJY*e-b6>g*6>Yj>hRz$atti8%g zRqtG`X?|Lx96naP88rEREi7vxgOUBMZsr@NBoU>cppXb!?=b_5^MZ3h(a{mnN45zt zKR^I0fcSr5|KF^!(q-REVYWVZ=e@b_;818KgFy6B>jP(o>cZA?^YVJ~akjbGn3~{) zwmw5c;knI?wM^|m41LUOyzS;)eCCZvA=7*WNJEZxx6`SElO zZ}#`6Ga*AUB94!}-ajP0_&!wIzL!8C#7k*T^!Q5|S*9WG0Ri*y@0&dsX*bq(Gqz}E z)s9RaV=sPt5hpRaXu!`Zr}j8Mf0?S@bN|hscFMu?V{cbh=0<^pvrj=8n=7MNr?W2$ z9?sGZgvfthU6^z)pLmr%*^<~;ad47X$@Y$lO}^QOYJb0!$(6Xrc(XglKMbX;1LdRO*6z?@`y)*F}X+c`%)ubHE`S&qN~`6BZ?c zLp8xY$SJ_tfxQo2ml7I1`-)G7dU>e?ZqB_pT;h!JYZk$0psXJfoBJy~MP2IG=XbZi zNUBS2{3H%>)=%rZXM1|_>ziUuoNlD({nF%{ex491@13zHB?k2iNAU%e9%9k6ezQ}vms$hP6P~EbojCnsvhrS56+ot)4ZuSHNckXxtB3{OTIEc(dZufQY<#6E_zprD>O- z^fjvE@#}K9y0!1xNlRwd3e$vxn6pom#feOX7Qk7FbJhHKdkXb+evV^$56#RZ&j_}- zaOR52apala7V6oV+U0ZWbEc=ePnNoIj$T{sf9#!(pxT^{l0^A}XeTSq?AIE#1Se;p zPj_T=>TMkqndL9$3Qyi3@d-t7OO&LDcmbfxXQ9P!${!Oe=j2}8Qo&1W(dAOTru!@2 zn(a?RdW!jP4bbZM^6?G#FYb$b5#N<|ts_+56m=lyPH;wfxo00lj7!lh`Q#&aN}wzlJX=xXASLr8aORIK zb#l336u%bbyl8FcG9CbUP2nitj+nM%`ZfpnG=VdeLLjRsTp;K1;keykU)-L7Zsc_1bzR=*vspeQ; zjBHaZ`+S?HOI z`%&JbB`#yi7Yy~nH+8Dp)>k*zW*P-;yq*S3>!@=}L_L1#ykIkOZDImItEeKL`2qKB zH1)3D+LZc<_>H8jixY8hL&hb=Mi$FgQoiL|&IM5qY_7N}ww-gmtI?zoGCQEUw9h>UnY#`GCj&W4z0=vr z-2%=tiI3nee3t2_e;t4IqLS_<{%EN!t(eZhq2L$PJ*D2eW0RE~I!+b(n1sAO zQSD=LZX-K3Nq)DWpkl@f^Fync{hudVDf!mEhnk)#_VP#^f?R)iTfsA1C%7z@bM{S{ zs6&!~I=SVUcj1n70lpTXWcy9cAk6X7L9zs6iuLG%C~wNr=k7*NVL`eH;th8u=dwg* zP!!H?o@P2DeNAxFwA7#RCRjHp(yrAWavh86KudD#z3yy`;(Ou%58s=+v}nWzWGE+5-_RV zIG*!~D)bgwHwp>l4J$t-Ug1HuXBg&)k0Pm`%^XXdDm!)_PP;Tz>QM3rez}38S5+dD znqd>w}vDYg}1r`>ko*Aa!_dOOXrrNb> zBDBjGZ}1VooT$6Sn|lb4$y$3H*JHs+(NHN(A10p%om=+`)yQvZu!LXdH?@OF0&`g@4SxceI z-*d#KvR%VB{DC2Y?0%n|R)kX}4S+0xo<8lM6YF*t2WP)77+685YwkM2aBgqW$VeXHa8!Z8uEm-r|%__Zx59pSKUUb47ra zNOeA>tgTBQYl#(yriE;LYT$A|HsSf98abaJFVxj>$|M))#X=`9bLOaozP#~HbRDR7GyXDv^tvGfUGiyB!Byo{U(<^zk+XGiGGddM97TtiO$@ ztj$&8B!b^i2oAm{MXcJSWI@G3Bza`cxjce)!GOtoHP@?JhuH}d zrD}3=Se1Puk3E;Rve8Y?pBFnEJZsCi9|+6%nGKs8wd&W2wMwbu?EAeX2q9(P zw`|!;+2uQ`r{?MD`ONqGJ3hznubE?(({;Vy=Xu|+Ynl6)kpSZ2SpWbT5u~vh)=XCi0z;!jpm4M` zLc{~*gi`|m@~1qUpfGy`meU$xi*!`rUMj2N=0w63xX(%If^?luBQ7H~y)XzPFFj+J zmpx1t&V5ReLf!*{OMpUPp`0El2S*o(hXVJvybxTx*DT7-`AvegSKwCOE68c4YruIL zjX`iqibx5=z~W+@(y}68Nm(%&2_a4~5LikSBq0ix6$Xn#a8C&k=Z^=sA_Xpz$G~kM zXI0gIl*9d{zAhJcDI&>vLKg}{I_{AbkQe>k z_G`?4Wkje#u?WSzIVlX1#XXI|QV=moh&WgnBn<(9@REF2{+2@*4M*B|{vn41=(ikr z%HMO~)({+uh5oOy;A`>nz+n&@GzJC5Dk4!(TZE{SqpiH?uSPuO&zS`|jdnm|aEFCZ z6qgtMQ!}3H+iZhqIl5q>jxdClsv@pI1c`)0KyY!84Omu2SVr6$Dl8#p0}_^zlCcq% zmJtKXLcwrJkc{j%h`PG?_J5>SMZ;Y8R?Lso@c(0KJo`UV>tm3(-45!2r`bFD|7(dq zr1?iFO{5EMj(XzFUEB=*wx1axod4eXSBC@g+rH!k#ke5$R-FR(U(4?2M)q@`+Vgx{ zy$~pDZ%0#v?JWufTwe5Vt-lWIr}Kvm`TshA-}B?y|GkF$WrX8@T2S9peN+8=5*M@$ z)*XsLDBI%Z#Q%1^f6x4P>+dDxMfbi19FTt>jPI>~AEUo!?LWs$`x&mW!Uznow( z=^y33Iq`aaIKK}T-WU6~vHaeOyKKN+v55Y>Wcm9r|6C6L<>1#T{g)Htry~}@r$I0# z;v$F$UqHl#PlI4g#6=JhzJQ1ep9aC0h>IX1d;t*`J`I8~5f?#3_yQs>d>RB}A})f6 z@C8I%_%sN{L|gGL5f?rUg7JUEMe*wL*u{O$0syWc0AS7v0D!~*0LZ%LF z4?M+rS}z-Gbmc#>HZK~i;dFq(V3A~@RMdR8p9=Zwj*$T$R3GS_^rt)TCN0!=J5tv= zluMg>NXiYs#yrUD{={1Q(Xhs&*}S2$YnstFO@#PsgxB1+?@xW%Dto`R=+mxgVnPcP zIbkNh@A^{%PFf}k|C`?;AqF5Wf+j0*;v&lZdy&}*T3l4Ca(ypiq9DUX|7FX4W*`?d zd30Z$EHe{@YJ?E2ckCPO#b7~5i?k%_4HkCR`eXh_!4oMd3^#91({vg4nC5wYTn;(F zQVs6>bepNtetal}L3HEI^x2KwJ6Adv$ra14p*3V{Gh`}XoyiHj5-x+7F6I4@5baON zD0MXx<^~oif6+Ve%(gELu>}zX)wbk2p099PSUa;w!*4 zx%=Te%~ok)=Q&P#dLYJ7441>Ij&R-O=07eduy;t7+`?mN>8rcjkGfkK?g(9(x^}-J ze(B`=7Ses|7;6Xi3bnH8y_BcREwl5!95WE7k?t$C>M@+a#;sQD)rk9p ztD^EECa~ug9bQSplOe8wg&(!fSeM^RN`W6Y109j-wOcgt1B^O9E`c)xR701qw~;=- z9?8Egd071XXtRDnYSasDmvM}G=~@j|4N_Ee^~jrj!%99cLz&7##8Is0(kvMH8%dPx zU)1OAmwD>Z7mI9vQ*x=xh!@%fSI>1RSGm`bc%d*yaH`H1Ar$65Wzoijp>9=p_G$A~ z^u$ou)hKa2@TyxDFVg7Dppp8)i~Y5zlRf-p{Bp+|XVNdvm?oV+hrV-L=i~06;pO5) z^FkBhvBu#{9^fT2SuXyK*KWmfaQW@2gVdVReJjP2RA#S8nm^n?&m+;R*>N=j4Zxb= zQosRKfw(&hFMRauN?R}u9bZ>QDU%9O&_QqSe)^UbM}@aO>$Y3R@(w0;#)v>E;`8jy z0uP(sLL^T2lWTLl#=;}8mN{e5r;K-&d#X+^J+{cU0O%82t z21(|b=7D9(6k#2rd0$1_HfY?utF#wFkIadmlP|E|&fz&KZ}W0-*!$e+7l!XQ_0Mor zI0%O#x1Eff>|K{?8VrmjNnYq^nWi8NwXCmZF=a+PazanlLh` z<5>ky?Gooh^_udNp4A*f1GzxgIUEksJYP02GV78}m6z6P(pp5zvjj@7xs^Y!qUQ;x zy?d}*H{dWg5OyejMZxv@r+`s?N0xJ+>fdRZ*#IE6T4$QuirOnHb_Q4gGMX`BZ2;Pa z7)}T7-p+EQ#)s~awp}0O+s7TQ4c_oKFA3M0bqh-AU?c;CYWlniIh=S$V&#eLJ%N)g z`n}jfN`Ra37QbyDxqWu5_ej&FU|Pzzfuj!}o5GCl-v z)~TNzoSjM*&oFs@^I!u@4I};d)>pGj%-+T2bvCI1HKuJkJ97-hLtc=Z#*@#6SeW=#%&e+x(Q=AH2san{2 z)|RQ#QE}m`=R_EXxzp~-LAJ3icW-7|0H@rua_@yXfgqy;7_$jz?fgB##aNFep0cZE z!!8*blm=vewycp0nbHWKg?1{mb3>1G`Kc#2xr7U!A?cNwB9SeabMYJs%bv5>`l_lv z_}E$ORF-m1j23p>q^D0TL2O61xJ)xc+O?DcGTlC^(vpGj`_Ej325kAwT_PD8JV7?c zkQ9j>M@c~qzFperyxrQbKn$q6O+-0W&WI%RXqixjq;@T=bO*WhRn*12& zd0LShAF2BGBc-m0dW${mWsWi5YVKJ$} z_Y>ONJV^k(x4$r{9o(OqOBXqt5SLf(v-D=;Mo)IP*=;q+FJ&Q%tlIHco6O8EP2WF} z7)!s~+k=VLah8j!ntkkib}(~6nu;30x4NJiEFtH-`q$H^#lq%qE|) zg>u+YRDL#Gxrzp*6;uh7ummj*x{j_iN5V$x7aodZCUheMjSDKvs*n&f4l$N=7X7x+ z2MHgPSXlW+E-=sK$x?A8I1i2XF!0GRyk$QBd~FU$TaWvCHTCd&KWyb`VYOUEfqq)c zHh6B}N(L887fXW@$fHb8GT;`E5=7jV&k`CrZ=30m4>SEF(b{?1>v+ZnNp;=^w~zw{ z=JxhbnRhe;<34$DL_iIdKdp*o#?z2|rZQoX(B;Lp3fu)$&6Nt1ExrR!;yUD3maymJ z@0)aQ=9bJ3FPc5lA2MASam|-*;%XH>6Jt4}6CMdvoL-}}XJe~Ch3Q)4n8zZWskhAB zlmXh>bc)%l+GJ??eg)yiWVY>s$gBPaj=^;3dg_pSOrpi)g-5#-dKk-Mn%vK=o%MHp zImP^iIl#`zMEdZ78)Fs0)cyb{8M}-V84&O#tjgfpbW1XtRcJY*hFs%e{EEbtW^*>o zK8cpZ3d)7&w3%sZlktVwa@;7o6h7&9sKjUG)77#V&DEIXedMF*^xJN8FPsNkv$HMt zk8*gNTGwk@+nk`|S{FY0Y(e zG%U`xy9b?3;cv1MA3mj}$LBv&3XPe53Uj=Bmz|%BEbxd5G-{*P%RbL8X1ml{T15$T zvc(aNge4a72%wLMCT>v^C z>~OC@e|U23!Gi99ikF$~RLWsVq2RP46E z$<@!(Dbt}5SF+2E#LZ9|kL9dm4(YsP(!*T1d`<1SW-lt;)gnIDslEJj)3FCvPxb@C zWt=)ikBqa^I)#D((z=V7_&C`~AXl|7jVs5S7(SS1;maF`vuAUXm1-}x6xxyi>X~i; z_ix-@dfN8+4yG;LcFcMKB7R&)|NSk!NXGlFCe2Z0kX5hKVS%O8tva#w*YgxUo>Vrr z-Rz+>4fA>JGmFRptKU3!A-S>0<8F4o!s=LZF8$Gt@Q154HMukqZu**rraJnucTuP8 zMUUFLuaj0JC|bI8%T5|I>tU)cbf}%UR8~esLQ5ulu)EVp*P?VFZv5d8)MsIF^>(um zgNtwWr+B}zFHN!G6~h2!gMq^(SI^d#tjUo9K>U(5kw@2ls@Xmt!)?FDlKiLRww7|6({Z~6h9PaQuLpR%i5kUig+u+Tu~Dz`OustY zG(QQj+WES8TCA2n;pV%@wzaHQ%9UZQv+)@m##4Y<<+|B7H%d!4 z+h%6A&fMcGNq8;M*qfK2n3)i+{S4DKuwASeszFMhmV*k01hV4>s@A_1w$ciy2fOec6v`=ZKgtYzR z9orxlTTu?rAkN1wK6hQL^}MM8@CMuH()6s0r-7#|zPdz4NO5iqeLM$|;2;ruS#9L$ zHyImy2C}a(9d_Sx1)X6TNRm0YZaQW$eQjg&ruUXs+6n#+O#5J=<#6J)_Q6o@2%j_2 zJasu_H)^`W^`t(PsY$8T_+z)cBuq{P+9W+>HPGNCEI#o^C>QKuG5Ln7_}e7N$2El z`?Up>e9buQlQrHXjJoioGlg|-$LQEh%Yfm-Q}LTTt)T)hF|XX$V}sKJhZ3BNitJhi z8!ewv2?et)DxbfFP#dO5UxSvR&=$s$#UFjwU9 zbh>@amN&`tOU8i-c6QTE{T`2ia3jih^K$y11B{Je+mSE@|8aOAJn|p_L=Ej4yhQIk zJE0!w;#V_Hw-#nZPp&BbS!+Ur|1>{IHJKjm@NiK@R{C4m%8RI%l`+zGs`lGm`ir}> zn+;SHcl^BIh{aEGA&W;?Qh5cm40*~6eDvjAjn|t6A*%>T{$RIiumE62=hRr<#+Q!! zmYJKvYB8MqfLI#_SxRR3CF##D8hznUdi2ZrU|!|LV$IfIqb;xDtolCfwkiN~$H^W= zE?^B3nW?KQ>`E1uFf_E18xu7O5qEwc96{Ouh#Xb|@d`N4r$24-P}dtw><3IZ=_=pQ zsWJeW9@K9SAMh>@BoPwE^i4uf`l_kui*W2dlD7AHlor&{pOC|0jD_=^OwLF)(~J~l zx)m!{ljRuwzHIS9(pB{i9;G8a%w9~K{!$V?FCO^}*dCP&PEeyui_2si%%JE5wJ~y& ze_rm&Jf|qNlI};kpKN9bsp z;v>^t23pQYt>vyRqp+dgG@sZRu?l)bZZ_@N!SYH4VGWc-zM|7nmK6vuv5u~J}&^U9SR&`Oh679ITT_xc# z_%^>pydUaYBhxEz%(i7iy(Xc@M#MOy73FN1fR!XI=(P}g+e7X$(a3v&ZlAj9#MIIK z?g`Umt+V_t;A>pR)3VA|IUF4A_ba)Tp@yRhhSI&JbIR7FiVD;C`7JLEzMzaBaEVV^GpFSuqdT3?tHHaMC7NsLD!WhY^)^7(t}V4=)2ZwEp~ z+~ZN!%N!ZP@(Kn8W>c4B3v!d4*RG!)(}dSe=}+q<#XU&430^(UFD>!hA7+DaJ&dCh^A-i&t{Wa`TdC&@sW z>%Mr9x?bonGB|D?ghIAaF=Gz7Y2C9tb9Vf#XU`-&4dy!)9!{e;oWHwl7<+a!{a4d-MRhZQq>uhd?8T`woXzUivuZ(w-!lDb!1ak7?l z=)K1;Lkl~#Gq}pdP~lN04~VCwzb+qTHsW| +

Wikimedia Case Study

+ + +
+
+
+

Using Kubernetes to Build Tools to Improve the World's Wikis

+

+ The non-profit Wikimedia Foundation operates some of the largest collaboratively edited reference projects in the world, including Wikipedia. To help users maintain and use wikis, it runs Wikimedia Tool Labs, a hosting environment for community developers working on tools and bots to help editors and other volunteers do their work, including reducing vandalism. The community around Wikimedia Tool Labs began forming nearly 10 years ago. +

+
+ Wikimedia +

+ "Wikimedia Tool Labs is vital for making sure wikis all around the world work as well as they possibly can. Because it's grown organically for almost 10 years, it has become an extremely challenging environment and difficult to maintain. It's like a big ball of mud — you really can't see through it. With Kubernetes, we're simplifying the environment and making it easier for developers to build the tools that make wikis run better." +

+

— Yuvi Panda, operations engineer at Wikimedia Foundation and Wikimedia Tool Labs

+
+
+
+
+ +
+
+
+
+

Challenges:

+
    +
  • Simplify a complex, difficult-to-manage infrastructure
    • +
  • Allow developers to continue writing tools and bots using existing techniques
    • +
+
+
+

Why Kubernetes:

+
    +
  • Wikimedia Tool Labs chose Kubernetes because it can mimic existing workflows, while reducing complexity
    • +
+
+
+

Approach:

+
    +
  • Migrate old systems and a complex infrastructure to Kubernetes
    • +
+
+
+

Results:

+
    +
  • 20 percent of web tools that account for more than 40 percent of web traffic now run on Kubernetes
    • +
  • A 25-node cluster that keeps up with each new Kubernetes release
    • +
  • Thousands of lines of old code have been deleted, thanks to Kubernetes
    • +
+
+
+
+
+ +
+
+
+

Using Kubernetes to provide tools for maintaining wikis

+

+ Wikimedia Tool Labs is run by a staff of four-and-a-half paid employees and two volunteers. The infrastructure didn't make it easy or intuitive for developers to build bots and other tools to make wikis work more easily. Yuvi says, "It's incredibly chaotic. We have lots of Perl and Bash duct tape on top of it. Everything is super fragile." +

+

+ To solve the problem, Wikimedia Tool Labs migrated parts of its infrastructure to Kubernetes, in preparation for eventually moving its entire system. Yuvi said Kubernetes greatly simplifies maintenance. The goal is to allow developers creating bots and other tools to use whatever development methods they want, but make it easier for the Wikimedia Tool Labs to maintain the required infrastructure for hosting and sharing them. +

+

+ "With Kubernetes, I've been able to remove a lot of our custom-made code, which makes everything easier to maintain. Our users' code also runs in a more stable way than previously," says Yuvi. +

+
+
+
+ +
+
+
+

Simplifying infrastructure and keeping wikis running better

+

+ Wikimedia Tool Labs has seen great success with the initial Kubernetes deployment. Old code is being simplified and eliminated, contributing developers don't have to change the way they write their tools and bots, and those tools and bots run in a more stable fashion than they have in the past. The paid staff and volunteers are able to better keep up with fixing issues. +

+

+ In the future, with a more complete migration to Kubernetes, Wikimedia Tool Labs expects to make it even easier to host and maintain the bots and tools that help run wikis across the world. The tool labs already host approximately 1,300 tools and bots from 800 volunteers, with many more being submitted every day. Twenty percent of the tool labs' web tools that account for more than 60 percent of web traffic now run on Kubernetes. The tool labs has a 25-node cluster that keeps up with each new Kubernetes release. Many existing web tools are migrating to Kubernetes. +

+

+ "Our goal is to make sure that people all over the world can share knowledge as easily as possible. Kubernetes helps with that, by making it easier for wikis everywhere to have the tools they need to thrive," says Yuvi. +

+
+
+
diff --git a/content/ko/case-studies/wikimedia/wikimedia_featured.png b/content/ko/case-studies/wikimedia/wikimedia_featured.png new file mode 100644 index 0000000000000000000000000000000000000000..7b1f89ac98490d8c64ec722375bbcae0d8ec9880 GIT binary patch literal 21322 zcmeI4c|26_+sDt?BD<_Dj4dS0*o`$yNF@6bVGK={k+JVvAwp6jOO!;GWZyz%%a(oL zLs>(S{W+>%-}!Ej=XqYwpU&%L=5pWH=eqCnzOU<^W6tXc)znZXKYIKq0088wDvDa* zdn5SskQfhqt#L5|UjY&a6+Ka# zvdPLEm2yLX0qjst=1?~~TYD_RO`7dTTm)$EHw&;qf4Df=NVA>W9}sGwt_fAZIHI5; z{GxnFxR5whOoAUSA|Wm)d=@GQgNq8lBn05Xd@wiyCV_y#p}!6`nWLa3wm+t3i4;KDqU=!iPFPR}|3{s}HH;Gmdkyn1L;mjm*MY$`QCI&x_CLoE?!UB^U5i*#GJ5ozPZ)EY6=I?f3la+)-}mf06C? z{FMDjg`X=V1)eKH!4YNdgmKitU~FZ6W|`)H5JD9c_OnV1%B^mWMBDFk<&hHjtLTpv z|05ns(cB3ov#;fYiSY@+bp&Au5n+Ug2>2@k2Ez&R)AKJe)G?N5EBC*}5JbR)5P~q= z7&xAPjR97WrMZ*&|0oKsFHRgwB*F^gXlL#ugSIobMhQ6BTT2Q2(TL;uJFgH57+Z`Z zI4zWnkd(kbHRGs$( z{(n2fKZN<)P%3CFn4#`CnG5FckKIfgb>q*i|LL$r|Jaor%pI|){hE_z`%lsR-o}3K zQu~@8wTm!E?(b(Z$o-N)SxO1~sr8R({Z{^xq5rQ7_%l9^{NKlLy@s;??+WT?s2^Vc z9t4ZAa&k3yM4ht+bK-wi?LXuGS@_ohqy+Xq0c_EKUW%Wse_o;g!lAPhVN;QOz?q`2LGl^Xu{QrzypO8-#m+M}Ih z1b>b8mu4K*{_gv8W6%aK6i_mvA|gUy{rp(=f4u`mKX&h)D;yMr!uEQZNTp9=3hqw+zgzIpK3zx=$_93nV5#c%< z;=-kIpnZtzKt#9>hq!QQ9B3cnIuH@A!yztQ8VA~kxDG^w>u`t*m&SqiA+7@v;W`}R z!liMbeTeHoM7R!zxNvD4XdmJ_5D~7!Aue1R2ik|Y4n&0OaEJ?+#)0-Bt^*O_IvnD{ zrE#Eri0eQ^xDJQ7aA_Q9AL2R?5w61_E?gQ1+K0FfM1<>bhzpm-f%YM;0}p(=f4u`mKX&h)D;yMr!uEQZNTp9=3hqw+zgzIpK3zx=$_J51(=pPRNqU^z^ z{#?Pw`&{wP=z)(0LXj$3>Hy%u2>^b90I;zKzJCS)XBYs?ngIafApkI7qOaDT2Y}=8 zs*2}y+y>?oJ-u|g3+sCatgQrv+{s89EukyK5P~#6mTx^~RGQCe3Gid|Jo37f1-g#t zSpTC*H5HBrl-aaNj0eP6D2PQU^C-jg}j;jm>AXAEMeW+zqOm!Cb3F< z&U4VF(Q|XOcw)JCkY+;Y7=Yq4Qt}YeLrY{wBv7E|FPL4hr8X8*?lA<62V|n=i zKGvECQD$l?zk0xqpFLp5uO3jculXU_*Zh?1YjA^k1f86qPq-T1@gdZxFpUY&63nk~ zXGl>P60YfyJzbc$T!~gPQoKCZ3a|k%AW~KiI9;V1FABRll)6f)w3m&&zNIcEd`jYR zZ!a9$K+Xu@;lYT`)+Wz=eM;}^6(z*Q(YN``$ah1oa6CMS26iv=n%8YPCVV-fuKULx zR+iruPn?JdHP}7zxrLYLI=LVu6mCXuKwM2uiyHEP;Driz*_B)mDM1EgW1e1$-R7D( z-CCq@(h`u1+LG@&{&C%&k$3Tt+1y-ukgo5U_SreRTR^G|bq!e#BY+Bzs|!z%#>!I? z)eaxCO}un_W~-Ry>qwp9_IEm8QIwA zTT>{9Q&T=@IAyfEawj1BX$QvDo6yd@)a1VI0z?Qx4+u98!1|7n|)TZn~Vs7>k&NQ%lNH zQ}(1$+DFmhhd9GhPj-AQD>|k7%It-w&CoGS6DzqNg!S|p;%Yex<)bVHqx~YLrC*t1 zWvNcs*RBNkRi{#P6kVt78{@$uuI_YjVj5D20iaOh9e34cw!i|FtuxZ@`?k0^ryOwI zC!-qh0|MnTZT3247b_aP+6ye|06*6ACg$>MH4&o=uG)@{0>-ZahKzGCcd4lquOyQ#TS?_EpkwN*}bFlbrj~RbeG9=u%vehS65K(FA{-zPzp>>{Y>4b`_u}dTbW~vS; zYp1$6jhD`mHG0O)Ww3D;@!~0^dINxOU*Dc?&$hff#~^8n=pK0r_tstx%)>Z$G4FNG z+c>|~oI@>YW&_Ek+YpkmGX0U(lM-jvbg-lDMrqU}4JX9-Rllp?FZwD_pyjB51twli z;;$OgyI(Gv7ncRI5p}l-*9W(e0eI*f244?#JOfa8fucx9xLZ#wd2^9vN&1CLA0};$ z1%|xzEhai~hl{>^yVSl)wsy&N3%hlv(r3kHX=khxnP-N+C4=7SX#2>1{|?v5po>0p z{y~?%wR?*ZRA@Z*GD6465)j*{o^<9kuH0Tl-FFPOI(;hSzN#4zqh84&;PTP1@QEr( zlKqQVDnOI;NV>h3%;x{G5#OWi{`M@O%c&9v}*_svsmI zmN3!g^0HWW2r<)q9YsdmqaWg@DK(MWcbA2Ec#e%CSl8_OWcP9VM|sP4iMy*Br?`>W zQSaG{#(@OH#I$~RRDe9IB&YYc-IBt@ntn8tx--P$q6|Ki7EWNyP<9-lvtipOOt!Y& z3bVQ*B5eNgp65vBwM-EqydFCk5M}P%lmVYxv<{SIxXTt#ZDjTMz)-*>A9KX&GiVG6_p`(%X;G}bzZJlznj$X z)ZO`aco6v-GbkK(I*^?&b7|OX1FwbyJR>>@@k;1Mn2dX#kjbx89z;?v?Jh zUE|*PXdXRzg88%a1K#!f^QQN9`*3@r6R1P|ieb%fIi1F}i=yINh@0oLjD`p5dWBwhOS)w#6f*g}T5R&*8KH}fRf zxp5E$LGXww;_`yi%PVO7{N<_>7#&$I@I`Z)y*s1g%+}AVU(38(xhr9YNkWlD0jWbB z=9)Dr3;Q6w;vI-1T%-bPlwtUx9_1B(b5s-Ok}j@G#522#zastzUH!$zh@lpq@%Y6I)J5Hb$oK%pc>p(DoCU-H}eOA3w} zW#2xkGH?ggh%Dw%ggsYf4E4#q&vzqEHJwAL>1!6t1Z*M*E*W;$k2CCb5a37jFo`ck z?>Jj_FDG)U#T3!ds%}Q%s?8TEolqO}w1|X^k z*HYOC<(@Y9tt?3NNsL(ZC^ajoS;nxkF%r`NP)0HizzilVgz*y)Tl}7%@;vWp4_AQ^ zKS{YWgje!y!pPu>YAVSY8jtc$r@HS~7hlpGLF6A1l!r8wV#w?Lp(`t}&^V zk2nVM7+pDEK%e5!YA}*Ccf*4R(wKx1T(zemI7*dLH#9zZj7|Q-oljO2udmoK*2Yee z7V$gXVM!p3iT?}{PKi<&i0rBDX20@4(x8()h}GyO{M}Ih^>coCOhJkzQE`irg4kOs z*;5UlgpZkN>LvHTt*d4Jz9lWV9qB za#Vg7BsRcLarD-jKZowCyh4ZhT1al^S>cy6wgi4-9qUygr{X?qROReF5A$3XgQQNc zx7zHGNtO=izItmkn*v0m3aHH7)~YMIvij6zfG>-&>E|N*SRc38R8M;CaVdc(96g5D z(DKSQvb)(oWhkdY&x70I9o^>e)-cO=t%|oh_%)evd0Wy@Af z4z^a>Q<|1l$3FC)#YMo%2MT8NrfAdK@y7-#V^ z4_g*dg!f!|xRr@Xa>xke*br)tUfT$jJuW@=y07CEsKx}i(rQ*!I0wB_C|Kr{`<{sC6evO&v-b7Wqs1S%XOaX9vPnLO!A(MN0sSF z4n=x+k}bwbHA3$CgB=U*)Fcv7@-I^el~|HCQBJ{?yLTMRj}L#Si)|*sT=`^FY&P+5 z+qOn;z2|C#XscnNiJov1XRWBl8@Wu<4k_I|=Rmd1nRi4ndF=vTeJ+`X6v|nq{_scH z{)AfEJ`&$vmEXQ4PpKKb9?Nns*?Z18mZ)SYKg{l7+?(jY%(KN58{Lmsyqu?5s`NJU z>b5!ODD~gJ+W1OtQ(Pz!XXal(8#&smP#wU$GSmx+tRTP5c0{)+$f4+AsTxaE`bzPL z_fCk$Dc-8k2+G zq@-22q($jr`VGm#?BTuz;Vn2q%ENoaYVj1Zg5f0Kr&vA5+%7Ozww+L8m_aj<_uxYW z@{Xs?i4^H~Z?UB&@FwMjoh^=O61j@ob#t?cy`G~c$5Mhbtffi4mD9ZDX0;647rYb; zK1pm|(Wm$R{>+5cZrujj{_5p5>B`~JM@wX3Ru|w=vmP(M-yVEz3`k#+4&wvFxr^1k z7rjeOAgXv{Ck&9H%6{DqMDr`s^!GUO9SXuM=i#C^riY^Z%n2%TWVh=Jj}gttN9sf( z!&bj+rq@4sFb|iG-}!_n7YxRp&+-x1V_0@GR(c}E8g|}D&aXo*b8c}?d05E)Va$A^ zYfX>DQ*FI3pR9;qI0h3DJTd+Zza?sA$1yohThsSUrF{x&D)E+ElU~IYpEZ~z>9a=1 zb*-8nF_@C_^%J&@GzBL-{@%L?3 zaa(-2iCOZ-V8|Jp=j)mz6nZws6Do2`RGHHt$qZEIDd)=VTNyEt#kv96{Ao-Dtj}|{ z=KEHCGXf~+fREE=PH))(P5LqNN~zY;ScaiZOyG(abFl)Hvu{&Ui3v%j6PQVj4_u*$ zo``}Cc*=uc!+g#C_2o`iOIORR#@y`eNebnqk;*5kK}Djf(H$MBTRX zmum(*owa6Bk9)ZS3sSFI`%9n{cFWX=#bkO7Hxx{@}vv^$|~q+0N#-8nZ8p3T-J}TffQJl`B{R~#8ktEgV zo1KwyqkK4;Lqu5ki@wX!2sw~%)iP0KWo7l?!Gi&)m8GTa=<7tR-qow4U0uTHj;F7^ zw^m#^RR;=fV(n;2Skuzdz}Qv0-*OiyUs}}OURhb`=+Lro-n_>Vs^u|gSIl>%#9{o+ z&8p4r-0xGwtRW8{JScKm7>JIJCM2cT(bLN@D$LBwyUX%0KK=r6V{^ETQU8YzUmCfR z9fWKnX>Gkv9GCVG24e2tU%o14+xHSX(;6we>(&x4vVA{M(iM@|y1mUWY4o|Qm9bNJ z!L@2@ILk1_H7eHz zY#8{Agk9$QjxdLKUVqs%<*_jpOQ$p*EcNw+0?Ef4ty0X9g9{Og&F8T(r)7AaXJlj) z7CP^4Z`c=mg0FMExtMF$8X6m^ZN1gboH;X4ik(T2!0PB$uDs{Xf3<&bsFrO^Kayr{ zdujfbzw~4D>Jtgy-R(Bny$bNi%9@{jkt?$|#i6?5QMo$ZqD_mc2_UpDmZUw(Ie((l zxw53BSb6q-cAvboG|n}HMPs`GVRbMHnCH1wy9i&@#JkYjsn_D`teL@Y zI2DSA9M!b7?>$2wJAH91w>e3=JTvnsa62sQ*z0dWS55e1bLJ?5k;@ack z(WwcB4(H*Bf-C&JiLN8)Hr2s_0aK69X@xeuBR*RT;5<|{(mP?{L9n5v^}&?zY_LjF zQ`rWW3JMCY#ar!_xh+r7@#;x!e7tu_YB}ib_~RIQditV*g1*rmU40We3a-UPjGZ0d z*|P=IWWjGI>+TGe++d&SgM~{jmaubi)x*NUaIsB=u1lu7J@ftfGtF_xdkhr8_2EpN zBO|TaCzKLd}oc+lK;8rp}Z)?)KkucYAm)&!1J|jDOXCh4C7WoMxU~6sR9tV$z2pw>S zmscSrs%?gsk5Ag^6V)YQ9ew@cq9Slt=;`X3A1uywT^gaIqpS4zHbHt9c!}VJi5`ue#OOnbLke}UfbUavt4n2`#1u&#$%?d^Bto**=dy;TU)0`NBi@w zMbzUR{bn@4V#(P#Q0~0#l@Jx zqL$aGiFky-1s>T8y4^iJdOmJ0E}~V>%^7QJYr#*4FXM07MrN(FbPF#f%Xo$M&aCR{ z>FMd}f}1cnJ|&CH$^BA!-_%rUI8fy)E08(GV1)U9K<{V0{;nX2AeBnaofz(9<(ZxxLKGTmF!ufI^|R*_VSXe0+RB{xk1v1$juY z>FJk}i_~6;%PB{k9KZkE(9lrS7c5zYcjp3$mKEK>wX)CYFG-oSjA7fmlEBJRh=TIb&kiRQR*#K_oq&aw39(W52suv6~f^7-s+g120P u6f1t*n)yH9fcf`ZGyig*C-Lf@B(U2}a^+FId;b1^yHr)uP|T4x3-~V@R@z(u literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/wikimedia/wikimedia_logo.png b/content/ko/case-studies/wikimedia/wikimedia_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..3ad5b630342048494a4ce20f52ea46c310c431a5 GIT binary patch literal 7875 zcmb`sg;!MH7dAeWj6)3|NTR26lB-$~#`gK>fO z7|~%30x`v@Daz{yr0-<}hEOTh^`epUp~<_}YEyQE@~k*XNa!>^xt*eHXQ326Bb2e8 zmeER4g}`AgLXtE-@+GyshfgMMF3uCSY%m4XSXusEEDmeLyIW5GEo=~4-+ys9@~yiS ze`9@n7Fo#tp6U2}{qcZ=uXYCZog52@FeVnx9*Rdw`WC{-kH~`j2v?5mAHYvQbR5qg9GTli8j?4>TV4G*?W6ZI&R27b;m$6U|N4R zwzxpKB1wgd0YO0^$VuS#uf2oDAqa|Y{+lg_VaV?UoR3Ku*p3H$${|k zp}3I>Fzn9bFXun%-IH!fePY8DH2qdT3d`up{Q z>k-^7sNdUn1~m6_R*FM;yJng9ns-)8c*t6CN^QRebIlDpbH&n`M| zzxy|%@QwdZ!X*#NNWS^ulkiV<&+phHQzD26`rDhcyJc_p=lBF%!?Aw1V^=Kkr{bK{ zNLaW~xvN4YSyJaJQ@h>qcV_i%7qv$}0_02ZML|$uK6V#sAt_~%RjP4%!wsjz#|Ffq z)_5XR5GA>TtIi&pN$aQB!&e-pGB<3Gv8Y&eWU$O}yZgR`PILa}bofIME3qk8p7ZB4 za)&>fau^Ut=V2c%CY&hGrww~1QJqL_cDlAme87kQ=x_JyNX*7Uygg%2IwUzX?B+OX zv8z-b+|^FOGVHr}?^494XxQ}%l5YS{3x}FM!sS*Rk}&mM!|LwRwcH)ae^-PcoadQKdn^2mmC{K>qxBG6`2)$k~}9@g zM<*f1{yyOW3)K4D#aNy6{lY(!?bDfGesyeusD+?DFG9DWR*%5b9|vepY|XZA$j(64 zlmfK|R{CswmnI?E zRw+98C=Oxt08Xr;IrxY9v8q;zP@ah({aVm^9O2m-ntc(n)N}qEm5G5iq|y?Tkw%lh zz=IN-f!Jj)x_?u9tlj)P(>UNGF`#!_7%8>?`25ZHXeSKX-@@bMR5us4;=WIvYqCZJ#$HMYTt+T27na^!g7Z7S7Ix z9hLVmb8nxYSW=9d2!U~izY{3JyBu1Xvw}Coewb&bsI4Yw`VObM2en_)L%MaeSn~#s zoSMMFmnZn=8uy~09B2b^k=94`@cxdTu1ZnKgx5TiUo02TM%pNqQLm;(`MYWX676v*YT3N__DzQA+r{8xWIa%5Rby z8t5o)a#{{2PQ5>h`ZsXuuzdfU`SDm=ExQv{pxYu?#tdgwi1ac8Mcjh5>s1rNt|2A~rK)<6Ts{qsiVpC1WlH6X8(GLo(ME6OA+WlU1ppe&x zv&az??vXbaRIkpkEa~>;(U;-czM>MLNX3n-I?v1PgxO4Qo;XXz8lVx zl}Of1=l&OWDFlNNQX=p{FpP*dm-d~w~% zv$f%xF}U2)a*{kD)rsBc^5DF!$*R@(g#4Dr%=4m#JQ-nIr!XOO;ygQrVrMa6n4|cH zH47&al;BgzRz<55lZbiszMMJnvbda{G2|}cX$_rc{wPAmioh2ab=)d}D8j{~3kts+ zlfDWF7@TbLWYZt-`-4DH#qCLlJ@-2j)hX*_p_FRJui{q*Ye0Y3Jo)lM2I)84dxNj| zb{R}O`%~+ii+%@K#N!8v*>Q9EFGJq1GDwSn`R&E-B08aM=tEJ8)ujmrMmYj7PL#NQ zK%z%8q{MQ+ftd}Ivfn`{NwJv}!Il0QdS_N#8q zE4^1NEJ*={lGoL!PZ0Mz;TL_44|sYbmw9oM@IZLWihYz#g-8GVm)ZE$(NLkJr|Ho*Qx85-XUo^f#6y82qPVMHe9VfigkdMy|#@Rr!zM5j`M;7Yh z?l?Lje|Ti>m)3n7dHq2c1C{G*i2Qgc_+i@5YNz^PVxDZ>% zClW2W1{ghC^FmiOCSf>jPReQ7TwO690W9_nmI z#;PhGkus`W;Ij9Z7ljB7AG#lAbW z=a+GbhVY+nbid9~_cZGZlJw=sC_$-Q5Tn2UU1TU%wv@sfZwMRX#TDi##G$L!% z+&D4NZuJrjM<6PwV17lc*mt)$vJ76aa#*zL z+R|xUQq$uP%FF;Q*MEB5zIU=346#-+{$#F(XJWy))P&o`?B&W_YX{ zPFp?%cd->wOW4QF21ERAsc6qK3FAYSeT@vZMDvR*9@|90408PR7n|sGr?wHQ#!$7n| z13%qZS;mx-o~y@xdQ+8a7R?{c!Z)oNX*3}I5NE?a!UI1o^BrBeQxo*av(O<(>9K(o zJB;wZ`Cl4oGU{N~>~OXO4xC2_O3z@AO~> zC431gnh(XHD=t>?#dGT7yBB2@xJ^tJG8{y7Gz0%-6pW@RiBVs{|! zJSLAoVFV-T{bAkG(k7M(?v}E#VN`ya8W`GHc#|0}h@h7!&edsIk#B#$_`&5#w$2KK z_SdWP{#O(4pn)@5R+Zn71>`s45UUA(cIiJUp{e&Uj z10vyXgOS69s7ess1x}UkwN8Y1b~04=gQOT_eO+~3v64-I1DdFUY!fGXyd*kiNGap0 z{#%XUk!Iitel*$1sy7fBhAJqrzxr(ujGAdQS%uq}$Q%Eblb}hEW>|}VCVl9W^R$fiY~5V>9j9;i>CcWoA4!Yq1)U8<({u7hp2o-}uL#px8A4zgFN ztC6 zVIkkMgF{)-h)rCCfY~%bV{uaj)|h;)M3YQrbM9yI=&~Vc)M>C)O$^P6BagfZtzLyT@f7?u*;I{aM zXnzb38B{2Oa5euR`>4b&2aAXmdY*?|U-$GjOt>@u*pTM5@zRT^_bw3644eA+(Q^1h z{D|L2o~`5`ySv@g{5~HHXVlry&+%7ROxGBUK6?c(*JKEIdEn<8^aDESt#Ra^F10o? z=Wi}LCE3Pz&@s88ed0X*xBm}s3ySft5$;sL#Yt)W^Qh6oD*Z-tjAlIHNw0gO-+o`h zOsH){iXLk?S2ZdBMXyDJ>z9a>Xkk!989TRIZ&X*1g{S8zM=yl8Ec7q?{T5q;2s`Ax zFJ?l?_xZ|<-wu~0T8H0|`Lz-iE=`Tyz%Y7U9?4=gXsWtu$EW*nOV54KNN}Al`RG28 zDpGUs?(jFdykMMZ7U%3?T3)9Rv_86C8Qp)Hxhb)*?k zOu#y`cF=%e5DF?o>Fj4h_=v2}?KLJe;OJ1+2 z>tR>I$eF8@?dKE&RuImO-~8ivvAq~GE`dQ}w)zi09SjHzf_|9ET509{_L%aMkU@17u$8)sygV|O>vRlP7xpw4;^h6(MlmDr$X`)yJkslkK{ z;jSJ|&Ulzw4G{c04g5+$%s&gQL8cNx{T4!^{pixcRs7;SLxmWPvac%*66`Qa=Sn{Q zDVt$K^@8T?7ti{$J!Y#0zu}x6g{Ioj<;Rmg+AlFtz|P9JxJOgXB$099i2_o-)IPZG zB#uMIW;fgFd9lNF!slndmvPe-bn2v@SD|N>ea)Xwcl#jOsy=IRq~9k)DPDc zMm<1CSHtRalzs09h9cNEmKZD!sw~fF7jr+Edg8us0{Vz)E-h_+2)@6&G+bPdO8tS> zeXsbxzsn`tPpiE~jE+>&>QN0VbuzcJ&;7o>WA4>ieu*%nlJqOP@d(?RC2U*#U>Y6= z0&PA`_r~cU5Fx;g9Sf>ezqYS8m`}_1bNi4^r{Sz}#|Z3l5vRx@V>9?m|D=j<91q+Q zP)eKR20YHMag7S7H&KDlE_3~V`)uj|s{a2Sx9!`vom?+7=PGSuX7`w0_?7qy^70Ps z`H?Y7>bFtBOU#?=u-NUhS$F^Dd1GT^Dk>_4SHw<#3dzGs=$|JgC!4y*$HhH&U)Vjq z+;6*ftDg;BA53+ftrK+`wfmhd4}w4-Rn^tAb*{7Jma^BbyK{}gVq*SN71mo@TM3n? zJCg&e>E56SIjfV+5j8cn|3PW={#M|W?wvd^jztzvl$vsJ#T4J1@6QL{T>LB0`eK0& zZJ0xVE8XTLds2E=X+^O!U)MLY7m6#ayIu_>3p|GV94xeIeR&R2*U+#G+Tu`mY7e_z zSy=%w$cBad{J@b`@P=8IsUwbx7mLM8`fUC#)M01s?*GIKT(h;#T#MT2;v*Z!T4^Fb z_x$kshK9^5ZIt4v`SKv%-rk1g7DE!~{l<-}lP&fG>&Y_nUsA`!`8A=}XMjabW#uNd zaYD+296U%QGW6<1Og;L={}=yJHY&HqQQiV)|8LTn!;2!dctcki9(_V@iyHu6Q+qpm11z`mqXApxxd~t@C8re2tSH0J3EN z;lqdSE&qC&58AFLOx;D@7ec11?16_eDKs38oBzj+5odR%RzoKVgTV*Ui@CMf{d#8e zCzuL1;P5*kj>}>jqnsd(S-o4vR~QysW>gvJy?=h~?dA1IDOKEaC3-k}rwPi&#kEm8 zX+bY!)15Bi9d;9kfg+LLUS@@D{?4(y-8SS;jvq~XsS742eju@t?%k6zP9BcMzJC3h zKbcorI&|c#l3RT+0PC5tPH>SoFkLMQdAvSkSZXrf7J5Bt5zI$JeAhzgGfhdtqp7XE zKV7}?^CQ>G49Qx6IS81~V(2*uvy2oJ9CGxN%QEaTzidZrAs}~hr}a0jh*mJL=$V1* zv`RV!+8HJrJs#8vl{pGA!zn`S0s;sBdYQ={Lb6 zhs#~@@$q|ECaB|}R>0XperLPJfATxYWG*@Ys%1!yul~~nfK*kD()lQpjAX+>x{@!Y z{&my4x8z{Af$j%QJN)WfozX;(1uQhVZ^;$F;X==?J243f#Ohm-1Vmv+URdmcM;D3z zyl?X20ebDd%GLS7!SN;PKf}q>_v|YC=1nJ`3@wbCe4I$QoK^ZK7v@LVHf~NQEfW(H z4@UEe!buq=pKx-PnKh0+D;tq-P1Jr;YFe+YrNt*8u#xU)XBW_FQ9e2QRVlvO@z>Su z9T3y~7w2$~;bxz$X90&xLbg4-)76eZgpo-db|lsa2nY}gdaiU=RaF7g;}&MzI3|!V zNg0t6I@>yxH8e~eoa6mBRgvR6_pCVhe4pe&6o0bD9rZ(2Fza`#!+3$V&GF*Nm$J!0 z@T4g?yzc;ge{Cg51BuQIIZ_KeoOa9tdVII?m?$wm>k&2M+AtfdIo|kHjBN@!MWs>~ zy+;*|DCs;39Eo8Ln+nTQ{<6Efs}MuVC>3zn5rOBIXof`A+7EnAPfrJO1rY3N3)KvQ zs=S3eO(D|gyAUKYD`>kYUn9BgYP)#gGhZ;kWA1!lg7UrQ9RVF1A{?Kci8tPvsi{ss zs`*UoG}Y8bd|QD5&pzAR+v}E3%pzb<77GdrhSNly&vvH)B0A4E`8qf_z#c&!MCIq_ zS5+p}{xUp1c+&6P-yq+^kx&W5>-i#@8D#4+Q{&`6=NZ%T?$^c_S#+DepNYu~;9Oy0 z;lF`Ayu5(jN=gLbzhv*FWMyTAgev_H7C!GW0D%J#pxX}Y{m{s?sZ;NxaGj=Zz5j4{@|IT(Ax9AvzynMn`5m^j z7VZ(>0P>)hxG!LGe}6x3Sp%GOy?};#ybO~UM-y|k7O9fP5u|Q6h3Av~k_q$ zZs+|M`OF&LLA_SGUlR~fKQ*ezmXCU1P^zx0`#~fQ5CkwjE}87jp@u-~pRYL%{?DvSdQ90GYKnCu7Y6 zQXf1PFiQ4p$~NPLKQ(+1umjL*K)gJg=OrX0fcy%0AwOxR>&A$MvnP6#U}J3!Na*I1 zr=6XhJ#Xj-qKb-&auBNE948YUo%b%51_qprHfm}=L(k{^?{Ck40RaFwaP;2f)TAo5 zpr%G2{BI%npcXaG!_ICCDB82E5+^#_!O@Q)e0r~)oSdqHs~v`hGhT3EtP-T}Z;k;< z0P1Go1IhPpIUas-(6CCXHFkBnYJ$U`~Ki;reV3@Ie32 z(0n`kUUbq7FT1KrS0`z3aB!!rd4JZu{gZJA0;Z2ZbkOj^V29VYr6#qKK_@WVr!QUv z1LXteT}z7;DBar_%YMKL1t%pXA=C+pX$0LDS^%TV+pbW6ivUVwCq2N8$ji&))+=C; z326Wd3gsrDZJkUYb*L_Oaj1ZGjE$#DO~o!Jc252FW_-I|kpR$k{uGThc&@_ofl&Eh znK3*p2sRO-&78(a)bh0izreY5@iX98s6LU{R*0 zrvYW(1TrpQX_N1cA5eHv^O=BPK>ap2m?Yt7KB=dt2T1W;t_m3lSX7z7qqVi4A5H6A zxG?|;e)aVs9@#H-M95?b2n$y$7f3gGR!xw;=i-#A>au^E3y +

CASE STUDY:
+
Cloud-Native Infrastructure Keeps Your Smart Home Connected
+

+ + + +
+ Company  Wink     Location  New York, N.Y.     Industry  Internet of Things Platform +
+ +
+ +
+
+
+ +

Challenge

+ Building a low-latency, highly reliable infrastructure to serve communications between millions of connected smart-home devices and the company’s consumer hubs and mobile app, with an emphasis on horizontal scalability, the ability to encrypt everything quickly and connections that could be easily brought back up if anything went wrong. +

+

Solution

+ Across-the-board use of a Kubernetes-Docker-CoreOS Container Linux stack.

+
+ +
+

Impact

+ "Two of the biggest American retailers [Home Depot and Walmart] are carrying and promoting the brand and the hardware,” Wink Head of Engineering Kit Klein says proudly – though he adds that "it really comes with a lot of pressure. It’s not a retail situation where you have a lot of tech enthusiasts. These are everyday people who want something that works and have no tolerance for technical excuses.” And that’s further testament to how much faith Klein has in the infrastructure that the Wink team has built. With 80 percent of Wink’s workload running on a unified stack of Kubernetes-Docker-CoreOS, the company has put itself in a position to continually innovate and improve its products and services. Committing to this technology, says Klein, "makes building on top of the infrastructure relatively easy.” +
+ +
+
+ + +
+
+ "It’s not proprietary, it’s totally open, it’s really portable. You can run all the workloads across different cloud providers. You can easily run a hybrid AWS or even bring in your own data center. That’s the benefit of having everything unified on one open source Kubernetes-Docker-CoreOS Container Linux stack. There are massive security benefits if you only have one Linux distro/machine image to validate. The benefits are enormous because you save money, and you save time.”

- KIT KLEIN, HEAD OF ENGINEERING, WINK +
+
+ + +
+
+

How many people does it take to turn on a light bulb?

+ + Kit Klein whips out his phone to demonstrate. With a few swipes, the head of engineering at Wink pulls up the smart-home app created by the New York City-based company and taps the light button. "Honestly when you’re holding the phone and you’re hitting the light,” he says, "by the time you feel the pressure of your finger on the screen, it’s on. It takes as long as the signal to travel to your brain.”

+ Sure, it takes just one finger and less than 200 milliseconds to turn on the light – or lock a door or change a thermostat. But what allows Wink to help consumers manage their connected smart-home products with such speed and ease is a sophisticated, cloud native infrastructure that Klein and his team built and continue to develop using a unified stack of CoreOS, the open-source operating system designed for clustered deployments, and Kubernetes, an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts, providing container-centric infrastructure. "When you have a big, complex network of interdependent microservices that need to be able to discover each other, and need to be horizontally scalable and tolerant to failure, that’s what this is really optimized for,” says Klein. "A lot of people end up relying on proprietary services [offered by some big cloud providers] to do some of this stuff, but what you get by adopting CoreOS/Kubernetes is portability, to not be locked in to anyone. You can really make your own fate.”

+ Indeed, Wink did. The company’s mission statement is to make the connected home accessible – that is, user-friendly for non-technical owners, affordable and perhaps most importantly, reliable. "If you can’t trust that when you hit the switch, you know a light is going to go on, or if you’re remote and you’re checking on your house and that information isn’t accurate, then the convenience of the system is lost,” says Klein. "So that’s where the infrastructure comes in.”

+ Wink was incubated within Quirky, a company that developed crowd-sourced inventions. The Wink app was first introduced in 2013, and at the time, it controlled only a few consumer products such as the PivotPower Strip that Quirky produced in collaboration with GE. As smart-home products proliferated, Wink was launched in 2014 in Home Depot stores nationwide. Its first project: a hub that could integrate with smart products from about a dozen brands like Honeywell and Chamberlain. The biggest challenge would be to build the infrastructure to serve all those communications between the hub and the products, with a focus on maximizing reliability and minimizing latency.

+ "When we originally started out, we were moving very fast trying to get the first product to market, the minimum viable product,” says Klein. "Lots of times you go down a path and end up having to backtrack and try different things. But in this particular case, we did a lot of the work up front, which led to us making a really sound decision to deploy it on CoreOS Container Linux. And that was very early in the life of it.” + +
+
+ +
+
+ "...what you get by adopting CoreOS/Kubernetes is portability, to not be locked in to anyone. You can really make your own fate.” +
+
+ +
+
+ Concern number one: Wink’s products need to connect to consumer devices in people’s homes, behind a firewall. "You don’t have an end point like a URL, and you don’t even know what ports are open behind that firewall,” Klein explains. "So you essentially need to have this thing wake up and talk to your system and then open real-time, bidirectional communication between the cloud and the device. And it’s really, really important that it’s persistent because you want to decrease as much as possible the overhead of sending a message – you never know when someone is going to turn on the lights.”

+ With the earliest version of the Wink Hub, when you decided to turn your lights on or off, the request would be sent to the cloud and then executed. Subsequent updates to Wink’s software enabled local control, cutting latency down to about 10 milliseconds for many devices. But with the need for cloud-enabled integrations of an ever-growing ecosystem of smart home products, low-latency internet connectivity is still a critical consideration. +

+

"You essentially need to have this thing wake up and talk to your system and then open real-time, bidirectional communication between the cloud and the device. And it’s really, really important that it’s persistent...you never know when someone is going to turn on the lights.”

+ In addition, Wink had other requirements: horizontal scalability, the ability to encrypt everything quickly, connections that could be easily brought back up if something went wrong. "Looking at this whole structure we started, we decided to make a secure socket-based service,” says Klein. "We’ve always used, I would say, some sort of clustering technology to deploy our services and so the decision we came to was, this thing is going to be containerized, running on Docker.”

+ At the time – just over two years ago – Docker wasn’t yet widely used, but as Klein points out, "it was certainly understood by the people who were on the frontier of technology. We started looking at potential technologies that existed. One of the limiting factors was that we needed to deploy multi-port non-http/https services. It wasn’t really appropriate for some of the early cluster technology. We liked the project a lot and we ended up using it on other stuff for a while, but initially it was too targeted toward http workloads.”

+ Once Wink’s backend engineering team decided on a Dockerized workload, they had to make decisions about the OS and the container orchestration platform. "Obviously you can’t just start the containers and hope everything goes well,” Klein says with a laugh. "You need to have a system that is helpful [in order] to manage where the workloads are being distributed out to. And when the container inevitably dies or something like that, to restart it, you have a load balancer. All sorts of housekeeping work is needed to have a robust infrastructure.” + +
+
+ +
+
+ "Obviously you can’t just start the containers and hope everything goes well,” Klein says with a laugh. "You need to have a system that is helpful [in order] to manage where the workloads are being distributed out to. And when the container inevitably dies or something like that, to restart it, you have a load balancer. All sorts of housekeeping work is needed to have a robust infrastructure.” +
+
+ +
+
+ Wink considered building directly on a general purpose Linux distro like Ubuntu (which would have required installing tools to run a containerized workload) and cluster management systems like Mesos (which was targeted toward enterprises with larger teams/workloads), but ultimately set their sights on CoreOS Container Linux. "A container-optimized Linux distribution system was exactly what we needed,” he says. "We didn’t have to futz around with trying to take something like a Linux distro and install everything. It’s got a built-in container orchestration system, which is Fleet, and an easy-to-use API. It’s not as feature-rich as some of the heavier solutions, but we realized that, at that moment, it was exactly what we needed.”

+ Wink’s hub (along with a revamped app) was introduced in July 2014 with a short-term deployment, and within the first month, they had moved the service to the Dockerized CoreOS deployment. Since then, they’ve moved almost every other piece of their infrastructure – from third-party cloud-to-cloud integrations to their customer service and payment portals – onto CoreOS Container Linux clusters.

+ Using this setup did require some customization. "Fleet is really nice as a basic container orchestration system, but it doesn’t take care of routing, sharing configurations, secrets, et cetera, among instances of a service,” Klein says. "All of those layers of functionality can be implemented, of course, but if you don’t want to spend a lot of time writing unit files manually – which of course nobody does – you need to create a tool to automate some of that, which we did.”

+ Wink quickly embraced the Kubernetes container cluster manager when it was launched in 2015 and integrated with CoreOS core technology, and as promised, it ended up providing the features Wink wanted and had planned to build. "If not for Kubernetes, we likely would have taken the logic and library we implemented for the automation tool that we created, and would have used it in a higher level abstraction and tool that could be used by non-DevOps engineers from the command line to create and manage clusters,” Klein says. "But Kubernetes made that totally unnecessary – and is written and maintained by people with a lot more experience in cluster management than us, so all the better.” Now, an estimated 80 percent of Wink’s workload is run on Kubernetes on top of CoreOS Container Linux. + +
+
+ +
+
+ "Stay close to the development. Understand why decisions are being made. If you understand the intent behind the project, from the technological intent to a certain philosophical intent, then it helps you understand how to build your system in harmony with those systems as opposed to trying to work against it.” +
+
+ +
+
+ Wink’s reasons for going all in are clear: "It’s not proprietary, it’s totally open, it’s really portable,” Klein says. "You can run all the workloads across different cloud providers. You can easily run a hybrid AWS or even bring in your own data center. That’s the benefit of having everything unified on one Kubernetes-Docker-CoreOS Container Linux stack. There are massive security benefits if you only have one Linux distro to try to validate. The benefits are enormous because you save money, you save time.”

+ Klein concedes that there are tradeoffs in every technology decision. "Cutting-edge technology is going to be scary for some people,” he says. "In order to take advantage of this, you really have to keep up with the technology. You can’t treat it like it’s a black box. Stay close to the development. Understand why decisions are being made. If you understand the intent behind the project, from the technological intent to a certain philosophical intent, then it helps you understand how to build your system in harmony with those systems as opposed to trying to work against it.”

+ Wink, which was acquired by Flex in 2015, now controls 2.3 million connected devices in households all over the country. What’s next for the company? A new version of the hub - Wink Hub 2 - hit shelves last November – and is being offered for the first time at Walmart stores in addition to Home Depot. "Two of the biggest American retailers are carrying and promoting the brand and the hardware,” Klein says proudly – though he adds that "it really comes with a lot of pressure. It’s not a retail situation where you have a lot of tech enthusiasts. These are everyday people who want something that works and have no tolerance for technical excuses.” And that’s further testament to how much faith Klein has in the infrastructure that the Wink team has have built.

+ Wink’s engineering team has grown exponentially since its early days, and behind the scenes, Klein is most excited about the machine learning Wink is using. "We built [a system of] containerized small sections of the data pipeline that feed each other and can have multiple outputs,” he says. "It’s like data pipelines as microservices.” Again, Klein points to having a unified stack running on CoreOS Container Linux and Kubernetes as the primary driver for the innovations to come. "You’re not reinventing the wheel every time,” he says. "You can just get down to work.”
+
diff --git a/content/ko/case-studies/wink/wink_featured.png b/content/ko/case-studies/wink/wink_featured.png new file mode 100644 index 0000000000000000000000000000000000000000..3c01133bef701699285d864d8b1924ad6e6b8e10 GIT binary patch literal 21292 zcmeI3c{J2*`2U9}v@=vnY3wm(3}YBegCToj?36JEV;lP#ktGt6Y!x9{vS&}0@KBN^ zTO>phDk<6HH(EVY&F}f+`^Wcu&&)Z@T=#Wf@9X}&?se{SCQM87#7-u5CIA4iQ$-oA zO?rPs`b%KgO8Tz}e41{&%}9MN7FYe6)} z3wzo-l4=70(#Jg=u{c}2E2lZ$ir^pvnyIPBbEsD`5g-kPB7?Tpv)*2Lhv zZE=zopyRSk(w<0?fIZ$7%js!v=iq|$lmUI!MUp;m6hlCqU#qy<%7ElH5^@@7XmKhK zo$;J-VR0}{L{x$kAt@{Zmz02s330-pBH|FJBt%3E3>85_C6Q1O&hIafEEDOIw6ldJ zQX8%GJsruC0a?4cIwBzu4-XGv4^d&FvlT={Qc@BEg+X92FsTOE#mm7J>j`#n;rb@> zLkEp_!8sEgT?s@7&JA6xInm8k1_atj^z-%ox$GS^G=8RXaQRM;#1Z0&b%clrLm~F| zkUvGbxN75nHS&*EPHNrvr~MB?itcy^*FQ>;`i!#B zpYmJae#SVuIoo{&ZGnT}?eO-bJT4>#B7X$nXian_x>ytclJckfE5tt~b|qN;1>lC~ zuVHvk!e7ERJm12;j;U{5Elq-=p@BT6|^kLA#6dgK+2;XhkATA*yA_0TRDT#{0;F6*Wk`i)a(vZJO{2Ec;)BHhRPC*okP*4&Vhf6Aoh$tfD z;bMx43UZ1HP&p`EP6_%$epBdgvMLTPu2=^g{(Gwlq*lYk5$0l&A~>)m4h98_;c%8< zNefE}Faisg#F@j*MMOkIe@Jf%|4kZ=cP6;wEtH&z_FpH9Bi7jkzcFxSKtD$A&)M~z z=+~i-B>dhcIT5%5S`ID_heOdDk18RiAfX^CE(%9OmE+4TPi~}{j>O_N z7A9F2EQ$ICkWFR3n*1$dBhj}Ce}HX?7ng+NEa6~DF$odU01zX6vXl@5i(6QVi(0}& zpytr8z&G?ah5tQ=hc(_omb6kMz)%Dj2E~Yol9p+tD3oL(p`>B@ttDS;{awU`XzA*K zb;iqCktUh!x2-|z$4bGeps=wOAUOFousDLl#_}QXL-*f-e@}q3#yVKxEo31-0)G^c zhe$&&YMS*ZBBT-yx82*#D`%`_`lXo{m4}_V3eoV?{{y<{sP7 z*i@5TgLDS7z`A1plgj40n{+I2NK2x#J=Rs0V2`!JLmVBfq#?fwH`V;Pmm?L3c0^}V zv+%N{&E-$Uo1!*$@r`|#v=CswE_ou-kl)J4>lmy)L(hEY`oi~=Buc4dW zo~#IBI-@p+h=e<7G)(o+w-zLS9TBp+)e-+Zmt< zq!WnC#wI{o48I4Fd;S<{_$yKbPHN=WbYCMkdA>(}n=G4-!e7_HZ>6MzHtCEH`FY6y zZHfH3{rpdU`1MHtld5D|)Ffm;D4SGVl=+Z(skq31P&TQ!DDxrnQgM+1p=?rdQRYMD zrQ#w3LfNF^qRfZPOT|S7gtAG+MVSwomx_xF2xXIsi!vWFFBKOV5XvSM7iB(VUMem! zAe2oiF3Nn!yi{CdKq#A3T$K5cd8xR_fKWE6xG3`>^HOn<0ikSCaZ%<&=B45y147xP z;-bul%uB^Z286On#YLG9nU{)-3QgKn{L*}L8A_GF%q=<{@*Ux70 z4y518dXRn;d#k8al=OpIPMor~1_0pA0|5930su?vr1wbxzzqrjOql@y$OHgjAMwhW zf0Rh?N-AhMjOW1HBrnGo`fkm&6KlQY)_pM+kFJ0p&&D+t1+wtDo5aw;cv1}+x_1Pz zDf^41-hujPU8tO( zGkbGp9x+@gQ@Q4Ae4#i=?=B4@-w=>jY?#dj`5JW`eBj~ko+ZQt#92T+0Ab1W1>ZG$4vH<$9^*R zWiwc8K*W6vfJJ70QAXB`Iu7G2(`OFs@N4I41|`DN&+>Qe37rf|@joJ{<(jZa6MF+C1=lp&A{nqjXMd>G;4KH~hdcK#_o-db@5ddxl&}#droY zeOEbSZ0Sgu_oJczTG^Rr+fb*^B71`F^ zy;E|Da{9s5Cun@z-BwmsOz^BWIbEmbeVCU!B*N?8%)d07$Xu|OZgPdcT!Xa5eEBb2>|R^Yjx{9kW@qy*E=;vV+@w zBjZJ!%;A{>{D6WTi#$=nMcY~;CbRcB4tZSRA`oONxV&OH`i_n-a;efu`^7D6 z3|{FUbcOf6n`a-}4QJEnepQ!etQ|J&URfFkmM2=!)AVEf)ke)L<~}|w;b3j6QhuLo z)l^e>Mj>ohqiX=C9l$Pyhc!h~Kr~0KR3MS{WVvNHIEhovb3W_f`l`IG%}QB)*vt_R z{obonq0_+Sktl_iF&getG?}4=-JG~ca0;|Yxh0~W2&M0L;YvG#!Uw$ADiEk;AB~tF zxO_o>V$It+;wVEz(cmlHvD}AD{zv`2P6eDggMIP%*39a_(>L=cuC)sC+150=rG+Lx zkLKMIR?^t$^0rp$%a_`B?!6}3mUa4L1-s}tCk3xI!Yh|TM!(P+$e4m8XKch6J|ViV zJ=u~Ul=6Wwjgd2LZ;el5_71ygCM) z4nE>-vTsO(vD%t<|Eow$Asszn5C8qn#p^HOr!-PzuNa5&-c8*?-{@9<9<=k!K)liC zPY&y`+h=9GO+OC=R%zSXm?JF$-s!N^Y3Yqt3F08}2THt=D-s;WdYyq9dMEX0`%%HW zz}9M~0G3`X;lUbS=pMJA7vj zCf-ZO@C)x*v}CFe!}wpfxm0$%;3?$h``Ki+Du>f(VEjSum>T(KmY`39&qE_#g)W35fe{t@2!Cyzk<{u*+q1!ODzaH^5noQ5nq+F}Tn-&k4sM6{^I4uhp>+(! zSAUUNL+YCD_2lC(#kx<>Ny8t=@uug6a`J-)OSyXJ)cpdG#%l%5#~16v?0gkWw&#i6 zVu8Nex(B^j&e|zQ+p@;3XwI0sY7kfonS9LHU;ziLdpByBF$ZUOM731=oLGIT9mq-J z$((hU4tZCbsPYIGGimIj#4!j5V$gIEwrSRMJ1UsrWhfm?D?@oUofa;9PT@(vZ@HZc zFZ14=87ZO;5Uskwth&e54(|j5ob_<4U1I@?L7X@^&o9z=rc&b+vmt1rn{S3&;;^@pgEsR4MqI%MoR^L{`?7G!?ofEO>{k|VT1l@u;_>BlgMW@uyVd%csm}}51Mhm< zqHI6BTI5ZWt1B!pvrYH&#OEHda$Ioh3`6f>8x(K0J2*ccAC33+MV|4U@#vY4darD%&(f#Bcasyv+yS4$ja^&hH6G#E=Y8s$xNN6cBy-k? z0uR7x=c2|cS)}{&%t?kBUb2v{!B3+Ztb#~wf66;m$jX6MCH?^4J`UH z0imOUgMHRBeb%GKV~MB~=l9s->qIDPDu2qeJ&Qwot%_*xwFg#gldTjqJ%CE>7IL&w zLOr+k_hCct$`$I=YYR&98OSQjDjEQtd!|G05tF6#Occb@%;9$Se(lW3mbS=}Zd;+f zHtkLD`p|9CeQ-bdN?J|n&MT(OUHsi+uT|+|7|QhywyCp)C`mfGN--y*j^;-n*}Ff~ z_En@OvrHu2INvauHhdaoY@{dg@ogKA>y;SnrHQIl-`cm%6Mc$q6-O`dNBFPhq60PF zv&1Xi1*lraD$_w19?(a>3v{S>bQ^%4yHUH|Lv!x_+D<=9qBkL0h5k#@d4Ff)k!asZ zTaK_7nd>qfBa82&IZruUbb1kW^;4aheXyT-q@sQ++UXL9!-{FJ?1|A%o@nq&0c4=k z*0=NF`_GEArqkh9F62Xo+=tKlEJV8k&aj=n0R*V+xf@U)ed&-SpJi%`{Zqp*`-S(a zc>+;33{0K}Xz9~^gZUs)Ye&G{dIABcf!*tlG4Nplf~YlW_Z|Sb4NU*0e`=d<8DDUxY=kc$POs1yw7! zw?~**jS45hAF8!zXD?(IJ-5C4Quq{} z+(6NcXVlf4s+AreyBhwFRn$Fq06aU=mab8(eqt@55|iYuBK{%Ied41z?(D)?!b2gs zXHY9kOO`iu={%Qr`M;YUE#OtZm(j6u9NKKKQb)UjI)O~^^k7E0z(Q}l=vRIKyb7b& zlkR-)JGpeLe@Vj$Ja@>PF{ND>AaHj2E(0`hYO8vcGDFt7UCfIsJ1}qPbOC8b`7V%o z&j%`_lB;7Z?^n%D)u&F^#Z;b29*|__R`NY`*r)pOsV8}lQnI@D*K#EJ%bqaT4I45| z@8S_X#Z6l`tM1<%uULLn17@+BytO#y<91YPLlp*vl&DfU7J9i5J$3Lxb^Eyj@CwQ+ zdaF#ME|`BI*E>emebw-hQbfk#nsq)NW`KclezXP#8fPn{KHh)pJ?Pbz!{xiTaa_=r ztIY09A56yOq+yt%j<^rCp7P3W-e-8g!F>34F(T(F|pn;xFs%?)a4Rq_A)<0k4P zcg_9S4x_UCr~DP4wlVR;=yq`j24y8U=EDVDY;0&z4aPfl=ImvAg6Wz+?`T|#{YOl@ zM(FL)Fl*YVvI@Rr$x+|uIw&#jiCbE`f>TWLRin6wZ`^v4*vIP&dL(xD#QL)UJbDv+ z6TI1HOT5a&3!FWK5!|geb)SzaH$gs#0kY0Ec|3zjjtY>bs;o6*?u|#F0PUFM z51H%biSxSL{`ay%_DQ{1o-aW*f+WK{fVQ_YwgQ=Sq92Z1oM>X)*2nK5wH?69>svgS zawZ~(PAl?-=JoUPJM9@!0>M$e2dfiat_tW%>K|LU6wXC^(48A1bHoX^FHc!LAY*5# z-lr5nrwmDjN2r$}s_#a0)bgIJm%^NbF;?ya+^?R7vy>?dhlMLayDna@?HZ5*V%~Ga zJ`uW7&VR4AbE<8;_T-CQ?yQ$*dEOHWku+}h zqdwNK310fqFlM)yQNzf`f_*_K8rG1yA#CmHxvKkYY$}HX9hYA0oDb+kX(X)ssWuCE zq`DjD^{27KMO4>hm_P)?8G=C(!)4)nI;M}chTCYys>_r#9y& z*gF}AkvB}!{kep$NF(&86#^QsG6&JpztQqcf7Ehg)my#cVu5g2P4AfcI}6yT%2~If zY7MT|iELlZ_M|s%KpVR}yqvo2VMicS&x8(BRwTpyfoXjr$CY;^J$)d-W#2!go$^aF znY+(MaXHxsd*!yKl=nQimuIvfB&WR0QqIS(-*7B#Z@yqxe=G9Bo23_iobPbY-JI6P z;YG?fS?cs(vJhLh8y;r$TrYctJoAz>oX%g#z|$w96Z!U-wfSgqXr_NU-)!}Gv>t$% zlVyPGoHDk()X=c@>TP*?L1Dhh@wmcJk7$=hgE{%;!l>bVNIy4&8bg+EmcZQ=gJb@B zbxD&doO?SvCX{EZWNTv-uS^hM8{S${FRRaq0o)U1?u(nQhO>bgp4-U|nH}~B3{wQ@ zI`A!-CG{WmH%cCwh-5Jg+ZNw6p9k)0V$N!97c)E%Y9eo(DthZG6Kh#r=6rg=sZ5@s zCq~iSLPmuW`8+G8tn4s(z#u^R9@9xvwiPa#)C+H?1?QURPl4gE2N5+~tmd>8w_8RG zeRs`b@|RnNA}+O)d$={;vmfc;J&@b<$aU(WVFPtGk#CU{60a#u!n%XVA5V|V7^ zZ}e63f~Y89(&P4pU1TgQ5|r8bJf{3T z0a~m``w$4dATV=7K!jo4mA`6G|C#8tA}s)+H-(bz=FY`v)#iMNy&Wxs)G(=7j%1g~ zw+%{1tU18g_Is_<=CGbcIJcc*x^`Ss_ZCxNuiu?VL%!02+e4n_+~{x=<7o8joQ^6~ z23)<~U6|1-u{Ul#d6t!#7mw&>k73%4VAwvBk68GOq$%lQ=LQ%u0;hxEcq5*Zlau3u z@y`N|I;W!2Qgkk89q9S+s87eXi)JX{GiWdBi*JOUamzWMwJHIBS$*>clj#&W1KCHs zxZS4a3zFJLIUec^K2WNuKFF+eU2bH1BcTacF}Hm?SAhqfdybaR^Naw-fL0;7KI&ve zq9KGC;HiI`mvoo#N{U6k4g5xMH`(i(C&&SubEQ9~FLMFq zIa(N#^ch5BHf)GHd~(9HJZLb_en5Zm_1%XeHy%V)44UA%*+HPhXd;ivP9LTyp-Xnf zjp-VQ*U}$Skvavi;1(}yeT)uyz3&ng)lcft#oJhh$)%o7!=!-Xn@Tm!R_i@n>{EwNTXAF;1taFWm@;E_rd(&g4@%ABTnm=%x-^@csrhLn(Hnq$yyC{_|VRaXWq_V@?mc~XKL?!$f}8L=$SasZq>a? zYJHm^NH@V~Z<*FwW|z;Bqv}AEP>Ehqs!QjXIs2XVcI#+ptGa*S^5)Co-)?s|Zj65` v{C>Il_bcB&Ts!|uv``be^j@X+qCkKUDIp;Vgd(C4RFEcJ1p(Ldp{^Bi z94EdJ)TG4kjzv3^IB;S09I)0%FDwpn7zp^o1na8? z{C7|erWODlBpL>gmr{`Qke0m)P*jqVmRGteBPRinfk-QWAxdCrIZ24LGDJxkA`STS z0TR8TJ-w8zboKxAMVzStZ(^}1WiS|r!%5*}rI2ViSXxO*2@H_|%g9I)EhI4^{#adi-&;VS{+9K}{Fx?V!oa~$6j)jca=xX% z3{6e{XHx{?Z)*(J3ih9P|65^< z1yNK`(vnw{mey0esxPahBd?^WqopV#t1qYb7grbQ5r}~KWB=lM{)emhkKFTyK%j`8 zbzx}lAeg5<8i@e>v2kVZe~v}&ANBs^dj4}P^8d&M6N3Stm-oMv`>!Qp51qrmo0mBG zyZvGQ#GXeJoBNyZ+f5P@R$}(FY=WmYZiFQ9j)p{>&aQh@slm-ROGjg*k|-Wr;{a+J zb1+gHzM-)-+Yn5{Kc5$jDBLzbGZz!JR|=TtW&bQFT2;%yNa?O=MVib)4H7jptS+A! zoz1APlN}l94Lf?(>qMv@QLB4hXTN>wbFDBtdv*0}_WNn;LinlE+a)ylT?bnWxeKI@ z4>SREEPdMK=ZFKyOhmQJcZnz|n))0`=mLnylb`e)(XpJP|2xI~eN;Y;e`HdD$mZE$ z>|~ln+eB_p)AL-*bS~O0kSd)~U%liPt~%FdKQq|%=SgZtn=yQRL;0^MUsu&cvYGm|HJV9&-Kfcq40r6{qzRO z2f|rfMpSYMl#Bd7YQN>ugj8xMrUpKe>+9SW=%j`h+!UrqLS9DZ@z&`v&=+Rp@>)XeP)n%XrsQ^E3itjTdYX zOds!GNs9x(hxswZeVxf4a{?}FX_xU1D+XXv-%OA)s${Z692p&#R;|pgMF*xA@hUxG zSClUwj23`Z>X$F?(iB}*@*=%RN+3P+f>>O2ixyEfzK z9eU$AMFs~Oa#X8zIH!g1Y&^pbxo-3tNuT*je<<;DaZjZVgHufC)hu?6%W~F5kWEsN z{9uXHO7>>)fG+QW2~P&h`<&(~j_07i zqOd^)r-b6VjIt!G*)7jlOv-+^>C{Js7A31``XajV`Oe4WHB`%81^j;eI_`Ox@o8js z{~~WkDeJEKVeA@+3~rlObs=$@&jTmtEkY%cMj?e-rG0Ejmsw=ga;T;YVKGmb>=Lzy z#csxOhSoOP?s(Ezk=#?my?<;%mqvBMVwBLZUc+N>@z%wxTQ3jphG<3^m03nxv#Qqe z(RB#$oaMI4`bX^eaW5ut)=}))6K-hDt~E*(c7I34_z^%r6^L^?_~<9>tMp@S?G^f( z53V`ami$V~@3P^oV|{iLgLHAWKMM)MY5XMj>JEn~AK(nDnS*6$IDb#MRi)lLYOOz6 zc9TMaFP-G2URC!|5vurV<~0AQWQhtMElK7xp8WO1=Y_3NLmBpav!>VCd>7Zt`r#C& zwJ+!oxY48iXp;ziw%=7k6+ui-=aEugRbG136diGK?}~A{2dPn!u<}oSM%AXHWefLW zRN?P>^Y`7x(%#^I?D5m#trwoLYeF;&&o-?nYwthrqIWu3O|7hLZ@+FB^3vA4LLW-< zJdH9cXy|3cS%+LTZ3RE)egLUc*iqTe%$A>>#JG{y_3Z=BI=BHM+nQ?w)gEV{sQ=xp zZ0XhSJXHyN)d)8BYP8l)u${ zoEc3JX!5!~Z<7f-ipL$j%@T@`*loCao(Kgh9Hk#+kgs3sKAR*YMPHN_b(O|38zHJyPAuZ~{!9hA7r#j5>lY38l_fB$Y+{wq(i=NtFCwWdM zk1kx){R!Y7+VhO+G=|Jmr3_;2e%zs_5tgQY3zo0+5}H5@I&Y76QJ8s6ABgxhR#YYw z3vWam%5_EVfD_353(_I8JAqv-ba4P zz9QQ};1mebi?<9wwR~*og>Rs&aUfi>n^E1Wxps| zReVeryR8O2w;HwTkF)8xMjCJRv6aPt)RYj6C!a3(RVipz9Co>k1R2_hZ&-yZ*+`jB!izRu`;0#AT=Nh}Z~Vaw zz^9AjMoTwq zd+`9Zc=D0(6ZhphBLn*`5cQYC&&~okp^WlQTQ8{z&4~{)L}0?;a3_D z(&D?{a6o>lAq|mz`IyJ(Mu~)agyV|25Vs&#I52 z?Mq5HvX9m%6Azv+$Lf8r97!L#152lh*QU&sHAX1P8N7{^P1)ZN|LU3ZIxCuPx!&*% zyShp;E0WO}GqQ%$uZ+_pRc+$5MpoG2Y|6^F;?=+#$pV)j!5j2&_BCShgO}|OHJWQR zN^dWu2WuD4J>%;3?#7NJICPub@pjA)>sN5=$#`wb-E6fn3d7&ZwGFGVbseiuNDqJ2 z*gLT=h|`-0W77)>hhz{Y3?s%XnC@Xvd4=RF09JxajW8i2Oc*m?$^VdIT6Ui5Y}T>BdRNYxOeck(J$o1`ieIN#pc_5(KI zTF|I>+eem|>ZKPZM_hd07U#TfSdxn83@0*bwD}yIDA5jl%Q!LBHjIf59_(|NQc~Jl z6j|@J9z2X|oWK6kieX2+C&;eWmWig&Ud)QmXDqg8)$N{2ajI*C*B2qz zWYcZO60ybE&%MWwdM|S4_*nB9F!8?(Qd{C99@}K}$+z~o; zquZWxNh1`)Sv0fvPFwR@>WpP^lY2*s%dw=pYl&r}@Wh#N%dY%blQwnf?rD5zDogkC z3koLAq@B-F>--&WS|n1vGR}%-V@ti2%>`O%Z1`R|_x2JyonJ568nS%ua=%sU*jmv{?1BpK1S({~&!2609K!&CE0RwP$9-zy8a>oa}p}uCZjrfZO{1TEj)Hu7Z zPh3|Nz=hL$?>fmv4is)?i56n|SZec=zY|W5Vp%f3xoI1tveIVX{kofp4|j2aU&^I&{6|$VV4#_-1p5ZtNivKIoXp*=6=z&ss8~uTftiv*R0{!qSkyJ_7nz(wrG9#{lMll91l&h-KZGZ zAD@{{`)!dj#S1SC4r1<{x*BzpRi_gL_EV~r*J_($KU&jYXee02A3K}1Y%gKdmeb-K zG9<{SZxoLiJhqP0b%{2&ba34;U~9pSN{fnHZSA*X>+mVXBO1>fm)ywT!_hA zZ?JLCUUy|?Pt1UyzJXPDR^D=dx1n#D{ORRf5tUzI_dNTk*l*aHF)o`Wq(wlHLJri9 zoo5A4AP#-=Dkj~TgKg99|vdyKFiKvZtZppYnHg@c*QiW&aVzV=xd(# zsx*0sN$nRxv+of{ei8m;w9TT@pRl)R@i=3$#H6XsC^1*jMf>aS&4;s&U0tW>v$5PZ zwH!W)99Nent%ow(nkhX;?_Ib))d+Q5@t5)*BLDEobu-Q|1jEI98$qza+0QLFr<4Ea zAM$&6H{{#UPLr;j4eVCWsf&5hb(OS=672M|Piw$|KqQLc$Br)#XZ1q2Bux-JUm3>) zUIkCiSQon&hOVx}ee8V&OuuM%acAqD?P^%p{%YxV_kHp6hNt32f=;v~ZuzRUwFF&I zxfFt)V7XwLjR?5&pjy4>PIO;KVU*}%Oi)X{{`*O7ns6S&Weu0UgiQDB8s@6CGs?hy z1%q4`a*R15P9jx(Q#uqC<*#x^_PBZ05)AZfPpUR~Aicf!-cq|$p_H_EfCnId({r;U0~_4Gx24F&wgb$)}-^g7nflJbJ3a_<0mnAOA1{-ktUP&y%{eVl2?-}Hs#YCJ2b{Osx3sqw7f_>>R4&GVfy;p z{5K})pH{DIJ^0;UXkt@1SILiRPb|(V>YK;aIA#}1KYNUJZO%Fxxe-&ifv0lZPCUkB zcx7o!ayZYQu|rC@TzRz{fRwa7kvT<=G)MtAsoHNb^TLFhV5=ID6!Y;(`5igHiuRO{ zb_p(#Ln!%~*@brWq2I>_frxcp%iWHTtpPW0(dfhsk?U2cLkR!>o0R|E<;#N?3BTFh ZNQ6ZA;4Y$D!1JqEBRw + With OpenTracing, my team was able to look at a trace and make optimization suggestions to another team without ever looking at their code. +--- + +
+

CASE STUDY:
Using OpenTracing to Help Pinpoint the Bottlenecks + +

+ +
+ +
+ Company  Workiva     Location  Ames, Iowa     Industry  Enterprise Software +
+ +
+
+
+
+

Challenge

+ Workiva offers a cloud-based platform for managing and reporting business data. This SaaS product, Wdesk, is used by more than 70 percent of the Fortune 500 companies. As the company made the shift from a monolith to a more distributed, microservice-based system, "We had a number of people working on this, all on different teams, so we needed to identify what the issues were and where the bottlenecks were," says Senior Software Architect MacLeod Broad. With back-end code running on Google App Engine, Google Compute Engine, as well as Amazon Web Services, Workiva needed a tracing system that was agnostic of platform. While preparing one of the company’s first products utilizing AWS, which involved a "sync and link" feature that linked data from spreadsheets built in the new application with documents created in the old application on Workiva’s existing system, Broad’s team found an ideal use case for tracing: There were circular dependencies, and optimizations often turned out to be micro-optimizations that didn’t impact overall speed. + +
+ +
+ +
+

Solution

+ Broad’s team introduced the platform-agnostic distributed tracing system OpenTracing to help them pinpoint the bottlenecks. +
+

Impact

+ Now used throughout the company, OpenTracing produced immediate results. Software Engineer Michael Davis reports: "Tracing has given us immediate, actionable insight into how to improve our service. Through a combination of seeing where each call spends its time, as well as which calls are most often used, we were able to reduce our average response time by 95 percent (from 600ms to 30ms) in a single fix." + +
+ +
+
+
+
+"With OpenTracing, my team was able to look at a trace and make optimization suggestions to another team without ever looking at their code."
— MacLeod Broad, Senior Software Architect at Workiva + +
+
+
+
+

Last fall, MacLeod Broad’s platform team at Workiva was prepping one of the company’s first products utilizing Amazon Web Services when they ran into a roadblock.

+ Early on, Workiva’s backend had run mostly on Google App Engine. But things changed along the way as Workiva’s SaaS offering, Wdesk, a cloud-based platform for managing and reporting business data, grew its customer base to more than 70 percent of the Fortune 500 companies. "As customer needs grew and the product offering expanded, we started to leverage a wider offering of services such as Amazon Web Services as well as other Google Cloud Platform services, creating a multi-vendor environment."

+With this new product, there was a "sync and link" feature by which data "went through a whole host of services starting with the new spreadsheet system [Amazon Aurora] into what we called our linking system, and then pushed through http to our existing system, and then a number of calculations would go on, and the results would be transmitted back into the new system," says Broad. "We were trying to optimize that for speed. We thought we had made this great optimization and then it would turn out to be a micro optimization, which didn’t really affect the overall speed of things."

+The challenges faced by Broad’s team may sound familiar to other companies that have also made the shift from monoliths to more distributed, microservice-based systems. "We had a number of people working on this, all on different teams, so it was difficult to get our head around what the issues were and where the bottlenecks were," says Broad.

+ "Each service team was going through different iterations of their architecture and it was very hard to follow what was actually going on in each teams’ system," he adds. "We had circular dependencies where we’d have three or four different service teams unsure of where the issues really were, requiring a lot of back and forth communication. So we wasted a lot of time saying, ‘What part of this is slow? Which part of this is sometimes slow depending on the use case? Which part is degrading over time? Which part of this process is asynchronous so it doesn’t really matter if it’s long-running or not? What are we doing that’s redundant, and which part of this is buggy?’" + + +
+
+
+
+ "A tracing system can at a glance explain an architecture, narrow down a performance bottleneck and zero in on it, and generally just help direct an investigation at a high level. Being able to do that at a glance is much faster than at a meeting or with three days of debugging, and it’s a lot faster than never figuring out the problem and just moving on."
— MACLEOD BROAD, SENIOR SOFTWARE ARCHITECT AT WORKIVA +
+
+
+
+ +Simply put, it was an ideal use case for tracing. "A tracing system can at a glance explain an architecture, narrow down a performance bottleneck and zero in on it, and generally just help direct an investigation at a high level," says Broad. "Being able to do that at a glance is much faster than at a meeting or with three days of debugging, and it’s a lot faster than never figuring out the problem and just moving on."

+With Workiva’s back-end code running on Google Compute Engine as well as App Engine and AWS, Broad knew that he needed a tracing system that was platform agnostic. "We were looking at different tracing solutions," he says, "and we decided that because it seemed to be a very evolving market, we didn’t want to get stuck with one vendor. So OpenTracing seemed like the cleanest way to avoid vendor lock-in on what backend we actually had to use."

+Once they introduced OpenTracing into this first use case, Broad says, "The trace made it super obvious where the bottlenecks were." Even though everyone had assumed it was Workiva’s existing code that was slowing things down, that wasn’t exactly the case. "It looked like the existing code was slow only because it was reaching out to our next-generation services, and they were taking a very long time to service all those requests," says Broad. "On the waterfall graph you can see the exact same work being done on every request when it was calling back in. So every service request would look the exact same for every response being paged out. And then it was just a no-brainer of, ‘Why is it doing all this work again?’"

+Using the insight OpenTracing gave them, "My team was able to look at a trace and make optimization suggestions to another team without ever looking at their code," says Broad. "The way we named our traces gave us insight whether it’s doing a SQL call or it’s making an RPC. And so it was really easy to say, ‘OK, we know that it’s going to page through all these requests. Do the work once and stuff it in cache.’ And we were done basically. All those calls became sub-second calls immediately."

+ + + +
+
+
+
+"We were looking at different tracing solutions and we decided that because it seemed to be a very evolving market, we didn’t want to get stuck with one vendor. So OpenTracing seemed like the cleanest way to avoid vendor lock-in on what backend we actually had to use."
— MACLEOD BROAD, SENIOR SOFTWARE ARCHITECT AT WORKIVA +
+
+ +
+
+ After the success of the first use case, everyone involved in the trial went back and fully instrumented their products. Tracing was added to a few more use cases. "We wanted to get through the initial implementation pains early without bringing the whole department along for the ride," says Broad. "Now, a lot of teams add it when they’re starting up a new service. We’re really pushing adoption now more than we were before."

+Some teams were won over quickly. "Tracing has given us immediate, actionable insight into how to improve our [Workspaces] service," says Software Engineer Michael Davis. "Through a combination of seeing where each call spends its time, as well as which calls are most often used, we were able to reduce our average response time by 95 percent (from 600ms to 30ms) in a single fix."

+Most of Workiva’s major products are now traced using OpenTracing, with data pushed into Google StackDriver. Even the products that aren’t fully traced have some components and libraries that are.

+Broad points out that because some of the engineers were working on App Engine and already had experience with the platform’s Appstats library for profiling performance, it didn’t take much to get them used to using OpenTracing. But others were a little more reluctant. "The biggest hindrance to adoption I think has been the concern about how much latency is introducing tracing [and StackDriver] going to cost," he says. "People are also very concerned about adding middleware to whatever they’re working on. Questions about passing the context around and how that’s done were common. A lot of our Go developers were fine with it, because they were already doing that in one form or another. Our Java developers were not super keen on doing that because they’d used other systems that didn’t require that."

+But the benefits clearly outweighed the concerns, and today, Workiva’s official policy is to use tracing." +In fact, Broad believes that tracing naturally fits in with Workiva’s existing logging and metrics systems. "This was the way we presented it internally, and also the way we designed our use," he says. "Our traces are logged in the exact same mechanism as our app metric and logging data, and they get pushed the exact same way. So we treat all that data exactly the same when it’s being created and when it’s being recorded. We have one internal library that we use for logging, telemetry, analytics and tracing." + + +
+ +
+
+ "Tracing has given us immediate, actionable insight into how to improve our [Workspaces] service. Through a combination of seeing where each call spends its time, as well as which calls are most often used, we were able to reduce our average response time by 95 percent (from 600ms to 30ms) in a single fix."
— Michael Davis, Software Engineer, Workiva +
+
+ +
+ For Workiva, OpenTracing has become an essential tool for zeroing in on optimizations and determining what’s actually a micro-optimization by observing usage patterns. "On some projects we often assume what the customer is doing, and we optimize for these crazy scale cases that we hit 1 percent of the time," says Broad. "It’s been really helpful to be able to say, ‘OK, we’re adding 100 milliseconds on every request that does X, and we only need to add that 100 milliseconds if it’s the worst of the worst case, which only happens one out of a thousand requests or one out of a million requests."

+Unlike many other companies, Workiva also traces the client side. "For us, the user experience is important—it doesn’t matter if the RPC takes 100 milliseconds if it still takes 5 seconds to do the rendering to show it in the browser," says Broad. "So for us, those client times are important. We trace it to see what parts of loading take a long time. We’re in the middle of working on a definition of what is ‘loaded.’ Is it when you have it, or when it’s rendered, or when you can interact with it? Those are things we’re planning to use tracing for to keep an eye on and to better understand."

+That also requires adjusting for differences in external and internal clocks. "Before time correcting, it was horrible; our traces were more misleading than anything," says Broad. "So we decided that we would return a timestamp on the response headers, and then have the client reorient its time based on that—not change its internal clock but just calculate the offset on the response time to when the client got it. And if you end up in an impossible situation where a client RPC spans 210 milliseconds but the time on the response time is outside of that window, then we have to reorient that."

+Broad is excited about the impact OpenTracing has already had on the company, and is also looking ahead to what else the technology can enable. One possibility is using tracing to update documentation in real time. "Keeping documentation up to date with reality is a big challenge," he says. "Say, we just ran a trace simulation or we just ran a smoke test on this new deploy, and the architecture doesn’t match the documentation. We can find whose responsibility it is and let them know and have them update it. That’s one of the places I’d like to get in the future with tracing." + +
+ +
diff --git a/content/ko/case-studies/workiva/workiva_featured_logo.png b/content/ko/case-studies/workiva/workiva_featured_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..9998b471049e5e561af77feca573b2eb4c77e95d GIT binary patch literal 5980 zcmcIoXH-+$wnjt+1Zj~XA~EzsNkS*oP(oKa2m}ledJ;+?bOZ(IC{?H=iK{#+;`uf_x2cjueIl#-(24;WA8OqxXuGrYRWs5BqSu%>T1e* z#BrQ>2a=N#pZhw0pojwtUd05jkF&)SP;O`vxDCz<4OGXV?9h5>l#RD*H(G{-73IB~%$GYQDSZnm( zSciXMrT!xprsRf3;c;#TIGpp})zh)Z;c@QvI2WLj(x24=@@k{39k9QIe7_R>7hklS zgD2Vs;fBKi|I84~;oq=;ib*IzM5L79A_z%h1QJC0DhMSA9ImJY5mknYg8stV{F~_i z!YcodSTKa;`z-zT{e0;JU;C#8e~U(=ESf)y0*lk<`VJyHB6BWD(3 z?*9O=W(Qgr%*z!D>@4}rAnlhN@+JW|QPR)ZKf%4uZt?OE*2kseHEU=q|9EfoHPgKh zEC*wdcCA-x-@E*?vr4n#>JB+w2Y3(sUa7tQ-q5>_t9Z+fs>i)hF867E0ccYu4H-vP z55yZ;GJK^D<^3cO@`#M`OmWrO^6~V@O30%<3P|LO40FOz)Oo1~%>XMyNGg|v!X9?_ z?Y)hSgFbozC5Uhk0?@VM3j%-hJP{y`6A|NdI7;D6Fy+^{N_l2#c-Y3c!&Q2pz07oU z$GFR^gNqR-t4>)i`Zi4QGX3Lf{+-OVNnB=)-`=DBn*hGWFLU=f5=(Aukx-t&?OZ}> z`Sj)F3}MCdjpK_&ZpT_P_~K6aweM+ybBIl|65+X?xY!%9GvkWeMLoOJLHnJ#p9zP) zM{2=(lvL)FTnUw(Eqpm|W+vKz1EyK^VBI+CNp<q+C->B2Z%q{exD{h&2YFLr-y5yaE3wlb6Eq0v=>8aC-KelC2WI7fPxiM6NH z#Q475sxJ~F%|^VtypN@xts76tI!Kak{<^Dw6Sv6cDq;k{&A`a# zKGpm_wTr2044y8h=Z?|qZwxQ6DZZe{g9Ozjjxqr%$wr=j$?t1spU%up7he@)Y zdLv1QN@{#=y6$U>V!FKV9($2V^Oi1ts&NK3G%KWAJwGH?NRfF<-{B*nH)EE%kk`4N z0<=VAszndball4-A7Z~+rdS{fU00{KYO~ABwR|~rh2M-Go-B+>aQM=1XjO9Bc~2oy z-(qkkPlc+wTB!n_q!gOzzd_RFUgep(G_pDYo~{Y-)7cx*%pmBDr_5xlZy!EI*ty-x5r1daZd%Hz zm94qg4`AVzg~{oyzo8eY?+zFNu?B7&|4{f=;xS;gx#Pub7g<1hF6A^^)v`hp|AS^8 z3LZCX3Cvh(Ha5C8SF zgGKn7iU?8{D<8_;HO!JtJF?0Yc#^jj1U(d$=4bJiDE7VQ3TBlY~ErPAN zhO#qzXL`Qz%TFPMyhBHfAL*(4K+Prp2!A|vELhD4uBLG+B+XLM6j&uTEQ8eDQ>9l;oJoo@mn5@QUxR%xR6;ot(`rw>Wjx4u1QZDfIN}_QI zPwv7}fI_sAdxFoL%J}+`kQ{yZV>Qa4=CdWwu?9lbu26H4SXfmiyQ#2li}>@H9_92Rn2&W#dUcZPM{DiU2PHM1=K}{0Rj9~vm-lMf z?Y&PPGTDL%;{-s?O*%6n1fr3u_8RRqhQh$L9huJ=GrW}bvyPp*Ki_%yb?E2TSB7_N zVCD>bd1~_6(%L<7vZRY9(?(chsBV7k#US<3^&G94B9HyoJc80ScarayMEOe?!Zx}~ z!NYmpv+|M7CvUC{`)!)Zn|SW|e@j^5aE{$KxJd>a99ko{P^GGYnWS@dHRPCl9PMs?jQE?L%m$ZX5 z?c3)hEKX~R=Q0y}1+idR^*!ePmp%0@cl-+N(ynzn8l&{rjT##IJV{s=UNuqAcqUM- z6f-`vC{PQH)#;1oyxm@OlFB8*kcEsp-9Ykzwa4UBx%*edPQGH8X5|l%dECa;-|?M1 zL-&I#u5uOI<3X%1Zz~5HPu!y8Q*J&jM+UDV12o^r5{^xEzrHH4063`pwt4x-J%ScQ z8X@(u=rH!!O+OmqpT2!@rP9ylcU?07j#C!yRdSWdfa$rfmp+gGm>$P(lj_|S7B|8% z$S=InNbt)!pPlU=-1WCjaBSsZ*@@(RG?=f=rKQ~3-F=7=7`W?Ja(djqAwc-Vs582#@ z-T3H-7ow$&l&Hm&pQe{UG3ViCNb$ip5p}r%VQ4i?&veOlir~1*Wbq zNF+Y^ydR8Qgi;mmEWImXXFe7mlGYxQ_ssEUJ$mzrT>9MBTY2x9EoIt^aIy%B(+6S^ z3q36R@d0Lq7O5b%ZYHWeb)SK+xRd;Y{xzj4<)+*^yVc?N+cF9FzVa5hPY=yZfIcbn zuxrTa{Jv}b$unkXuI*Z3k1GI)S_lLwb^RWIO;4<-mp18X(80<-+cM1eo7srkx>WvN z%)K!_ld(s6$lq0*xaYjQ?Ogj+KF!}Nb%E)rjkT_D&_!m-am5;qWz2`l3uf{M4@|~= z$zERX?!0BBwbSi#kkk|2Gnp0xeY{nIc2I) z(V-;|8ykh`c4%QMY!EiNj|zFQOvK>uWz5RuAlO^sQq1mUV-N;J)D< zOsxL7lgRYF8_jx?eIM;;VJ25igJ!$f7OH+M_Nm_Re=E>bDVufJ5IAfw*G zn0AzaA4qs;CfDH^R$);O`*?HdvYNb@jJ1J_2=SM>g|knqJ=e^kNU z^=qbUJuqVm#VhR}+P8KHHDOxS=V~=pdb~Vh2wIjg1$Es9JpRkoAKuP^dN#Q_$yP}5 z2E&7LdZ9+`ObrR^L_Np2PS5D^vTd}wIHy#e*^9kJ(Ka7YfMPal<++ii`;R>I1_ungTnj$kf{dPu;^TIW$8--&~?| z^|ik_ewo;YTAjt;7mS!ikwGK*2~tgELV-^d8MDZ5R<1Dk#@jT zl_|{?X0A<~ZjLe4ma?0DPTp(SuTP<$Llryp$oX)Tp>^f4^XFW(TA3erkS`~s5c_nE zwHiV98co@MWWK*;py8l+d)9)I?Tgo$r4y28NiML9&zgpC==j*Z46R4D~%(CNagw+kU#`koFR^&6hQQe%cnCI#B^;m8foJ(#;ly#t~w?}p}TCNNS z-(6vw@yy+SHjPtJGF_4xKnm1p`+|=eW+#t-z(?30<*<+LXCOfu;)-J9qcwcRO(}A* z1DC?c4na_Q%{GGgot!e{N}|b`Yg(3567dAvrQD>}xZhLS_VNjBOLk$qK1*Y5S*EG; zu)@m^%#8!gNhoq|p{s1>UE)m^B4^>Hvr5bHOOhe+2eHPJE#WW;_N6k9OiuNP1@kQD z*I3$&F5jDZ^~#77-o;l}UO&v>P1CB{xBS9I&F4pZ zhoRNB53i29=4+C2^`m-NOm$61qvv@9rrVzhKoo5f#PpUCn+{^%M-MmB3!LTRBQ6nQ zr4>A+hTEdv=CwW7x*0hmpIhb+x)!ka&t!cwE+3qc)-k26Xp1)JABwoHyu$+wA2hpQ z^%bMios)aLKcxHSW7%bh(xC6Kc+C3yv|D-Is{Lt8YN!g&pEOAi)sj`fZ5;7Nj?P;QCd_`#jBY(48!JN{eMlCQ%lq|8GR7QQ(Jl(PZ)IIi`&IkWj!!2QjT24eJ~StC zvxYok+{v4?o($;*6E~4po#@ysMp(tFN;2jciFa%W&6^ZKJhgQO`AT?rty$u%hbaj$=Rn~ zk!w!=`yXU-OC*J!YbH^#W~Yy2G}$hBl;kFbhVvg~E)L6$+3gv)PwR?)PP?)DG3x{V z(Z!Q;M~^E6flsQ~*xPaR36ppVuyB@nuLS=>GJxhtuA{nOy%ua7>wo?sy*4V+vix3c z^>!wLz&F41a7dn;X|4Q5$U@X=>y@3y{O&<|c`aL*r8v#)Dpx7c_@LF*vFUe53%05y zfIGjVOJ%h$Su99y7R?>B&2Nl%)Fvn$_|$dF^aq#s!Xo%|BOj}zqMeOn%G90~Ipk@m zq6paMA;Ss3buI``Cqs{wG~$-~w0)XsdT5VmCWH~bU^%D5)QehAxOtBRN22>dT;BxK zq07o9pc_&et(Lru@(n}mRc{pCFq23!J<6!u{8_h~%<7acF>JRet?XLIE@kO2@!2rp z5C2MC;*ULQwfUt$HiI1%h0r(GYTgrqcayJMc347rzgMlq!?6bLGL|~Fvt{cfDrdO~ zgGO?eR~GjamAgBfcB~s+I%&R7cJ2Oj>M6@iolVY~*Q>>wwLMjx5eLAAu^cv-RKT5LaZTa(+c_fb0G=Cyz zEkB&P`1<{P7VG3cjt+?M{|aIa1EpS=`1|rcYISyQjB`G*B5=cHyb^rwW#L!~0wISr zQPnKcp^HhVI@8PsXfq9W7)^+Gu}L#X4vnz~9@ZEEzyRY~G6%ph2ZuKjvL&+@ zyZTKpGzx6{k;)}8(kdow>>)$AA9`f$Cm6hrCd#CHX0C+U@3am4=ijP>L=vf|ryp?-tj|c3c*Gpw~(~C+`8Z zCxC1UO_FvYF<6w54ImUAj|JT2hd1q?fKGm5>mSr59Khke2T5?uKPqfgd19NC--ION+vD z=KU4#eYi2_e!O$f%sDq!Uss)kke(0&1A|0UL&fl4UHlhe1bF{)u&Aj71_mRqrivmM zns<^P5N!N;d34Zcs(aZV^$EyYr})7Dgf3UD)3&bZ5HzE*q-is*4XHJ(AGdT%kLXzF zF>RwZnE7Z%<)yq_6T?nGph_K%qeQTr8I%$5l$E;e%o)x|ZnO7rzp9rZ}r`wb7(fI@2bjrf#ZGdjIs1I#Z1TXkkUKqwv<-MGUj}bNLJo{QoIvqE1OqA0`b> z!`cM;yq|wAL2|Ys)a%Ul$<6J+XJM5568`# zCn*sisbC({0?sH@8gak>5Uz`XuUQg+9gdvT|G$i%AMc zLmeAWffNb3Gl(~?z3V1o7izR>%xfNtU#VZJSe?qVpjX?NK9NICoR|8r2bM8cB*N=3 zN^G=B@@hpO&-Lh z&fwNj=eYT#vYL}n3@UH_!n+w^ShD=n!7R5%Vve6B6*jFiQCKe5-!76 zX}m)lfE)9!8z$REI6AVP$cVG4U)hWwyBB_fJZ0xuBiWDzUWoCw=(9h2h8)-&w)vt_ zcWntg`-@)?MsiG;wqHA++d>FrB_E2%AI9m$$@{DSPANc)0Q=@R!ij}V2>=&-MH4Ur zT`m5bQXfi69uMu?Lg4wN;48p~Hg{kH2CU^T;WdiVf!Wsp=nphDmDzs-SABr6r0Kio~dS%Ke;(MGhvRR=3n{2ANBc zA-lvGEJRhGNp1VYIP-@96&(%H0q@vAHpHTsnIIc%r4$ekNl)9cwn+(Q~+YnABp8!!f63f#%@6#Cq? zbxY!EMDjOjC}dlrPr^z$Xhy`X!8sX$q>~{&nx0ETI!Xg01BI*b+k;+hCaSMDSG#I0 z8HQ3MDyQj%3k{eUOiOp)bNmOeQ5`naNV)XvA=J?{Su}#=QQ}{B( z4fP9I;){lHm`koN`N1{ zS5-DPyOYM@EEKvSa2G5z5-J>`*c z%3u(wh*G*o1y$O;>m%P=m|zGQ0JRXl|E&Gu3M}39YT^4$inA?-f3B8H|ik1ZX3X8(xdzOB{IN%YOn55c~w@ zV{oZHUipzfkIl5LYvBQnCC-c@tg=~`{&iQ5c}JlUY2p-(6A( zg6HZ@t$HKhn;8m1a(>TAJ?+I0+L0W@y>eGLQcte}f=!3_-2ULLW7^S*(VzB_a&t4# zu>=)aStJ3a{N6bJ-Bx4|F-=#>?QH=?$V12;12%u_dYhZGBSR7O`ln+$~U| zpD>TOVw8FUYiN!feWUtBG1OL6g;Ux@dqa&H%tUqw8QiYItr<2FWrPRANH=#Uyf_*r zR$Kr^jrxRovF(!h(X7M8hB(^D)fYZV<`?q(U%mt%CBBwjA)0UjtUk&5_1;&ts#Y; zFrrrU`v*uW3!{T3cA#_ze(d#x4ryuP`zNHV5KJ2-essH1o{i%GGg@tbKKp($YD>Ey`kiaF#YDMNfbt}B+5wRwZ``OZ%83iNTbJ(V0 zWqA;=F?A{z-xa_MlK0cmib#j|-TsawHzBbJ=3n&%?AA0wj$DoLiIbyZDs6_d?%n#h zK5r>lm^xbgQp^mg^0n~B1Trbf$x(Tm^9Hwy+Ithnm@ReEUH;@we&OK~fcAc+bQVJB zKMfxbM#ApI?!Hjw8I7{09_e}4-O4MjN;CBt(pw5J7;Xi{jdop{Ae(j)ILSks@Cn~RnB(K&NWcPuNTI^OKY=3Sj!W=0N_YeXx^}VbzscQL! zSFDvnqB2J2mNGD1md9=MDEpHy(X-o%g)~cTF@DP!@i&+y*~M!(N?~bOdw&24aNMeBzuzhv2MMmi;$8EFiJBVYu!}dyZ>cF;Npf9rZD>W>*ExEgr{mEy zKdJ^dzL zK*}vu#y8wbw|*bV5+Sk)(2=Ha5|5Y04mhHcQhpG({b>*ay0yaZh`fK~j>(=@JkS;X zl=fNgWtBQJ;z)3$8-isNQc_v^@qWYj;;XdzYd#>zM zE;Du=ommj!{r6ed)(7B#Pn*_SwAPSvUwIGja{#P3)Lj}?uX9#yw%vse&!MKPZEgcH8Rd)B2TbHk_vGCQ-=1e zUvn;n_bU_)&tT49N;!(gx*vF1N|Xn^ViPznPlJ}?**$Yq4rs771rTLASV<|G(;tNt z5to(`n~vXr%y@7QD>kNRdNoa zK?8uwSYt5~6iHv!vA(7aJb_eEz?Pb*_&NlyOb~#qs2cdXIyo@m2e~hJU5cD3Buluq3_06Wwu^#;Dwaw z5ANM8+SX<$JequKRj(ZKY#18Y#d6FI)_|OhM{~cay;gk!bq(WV@u5=&<4S!^KHOJCauC?NoxzIJH!Th3Zq*l?+ z3G^2iAdXzj54^`e)+&xX3F$hDSqB?Vp5U1genx z83_Rz2a^JJy1Vc{`ePsF-nu>i(V;cI+c-fd2m4yhjb!kdAA7zDxwoI(OH_J1mZfr@ z$Acf)g|rbCgov=y4`Kz9Di=s}NaP1S;!Lh$K@YvtnjF4OpD=9HtPog-Nym>3(dCiM z7$i6lG0~tT>^<3b|0F?fa`jWxtzVE}lMmb~=M@&33XRO>!|QXL32T&W8Xx6(SfZ23Mcm}Tr}|j$Kid8NTQ#zIFEl5aHZPV4+zw4> z4i^7eJ<77gM`J`0xRWG@H-?SFCdtlAH%?S_Qos@9!gk*91Q_`=4SAw=Ev0|Fk;>rq zo~BP)2!dUTe?`Ho*U=&y@>M%JXtZK8xMWmH}H?; z!P{(2J$5>>xS#A({st;8g4+xD1(n@ba38eo4)#L(mFMXKYgjv;uX=HwE_1U{Y#o`- zw0W@=fTA^vUPa+v#ljy&0K(A>N0waMF+|Bik8l`Qa=NB0DkEHuC<>u zjmpc+4Q+kWKVD>UH3~&}H+y6Q;-2c6kIAM+doS$U3#wd>mFoK^Lq5}*{seS`Qg&&f zmj$0sR*b|=7CEl|qi)xw)Rv~n0~q$oMNvOr092SHnG zZPTIATF7+gBfsdZt+SG$eH*}$!ns3BHjVhqqAaz@;;rMPDXt_86(&Au z9l4s2i{Q6E%uCErU;F z*=ha-EDHdn!ftL(&Zv_J(PhLg{YO@-oy!7hYN2e$(e-Jy<~ff{<=x`I>rJysbf$AdFl z`Qr9M1_E9oPxh3jQ|3ASFcH$V#^R%Y?Jksf>|z(s!ui!v+3j=SL@GpvR<%6qo5W=B zP$yQ7a4^4zVV~CAbQx zT6a1&dIqmKbZ%&8Bj2=lJA=_Mp=1u;qA!L;1Z7D|%NR4ls@3M}Ws2U*`?)bxs|^*H z3){2i`HG41!6n<|MWvYo2tqj|UodGzD_;ePm$1%$Sr1Pq0MwIF6MW>bwE8itK+VhT z`KnA5+zzWEbm)J?th2%|39%VuI$B+Nc8r?yT5=ej@fBk2vs^Nud{k^x)vdv)|IuB9`kFQOI~Pvd_9Q_8FX3v zY?{VIM_F|Kp0ZF7k$npZZmT&Uz&AxDaZ8*B0HC4F3!Op0avi4YZ-6RL0vi|{Ygbj6 z(nw!Ao}ysW@Dzu`sG(^hm`Yl4+1Oq7Pf$EW_rbbDYhSu;6o(-f#KoA7`O>B3IEa4V zM%8(Im858SaUt5XdGjB};^(D|74#d1J>+|1LR2838{JFw9E@WPnX3NJk0Dr9V zPR>kMe%(gMa)kWKEro^V9Or934Zx!AKg=Cfyw;caqMou?R(&(4!YCEE+PoCz3DweP zI5e_}{U^c9E*c6umHU-_T`$P`E+#(Mh7dOPFVfeeEjn?VkBRPlnd;1mj8$tqLVVUE z0AoIbNvSmp(2)Uq6^{YW8~<<9hr(B1P+tyBaw~yYTdjWjn|sDGooRf>JmP?5A?=fA zi3*gNyxbv64KkurAHd(oUTC}H`&@SFo9w7}0Jf7Kq2f$wX|%n!)jcDlHSOL~@iMMn zF#nc!zQ4sYd3fMdwe%`>1QygLIynfVNW%S)w{A3aP8)yk#Lx#0F51Bf$4H9s{Nlfm zxAJJGH+Dbnd4kV7L_h2OrMp@BJ;3J^#4LOHGPoU3qs2;zUy2yo_Bo6tLYh60^mP(< z9D~)1KRcmd?n|v?;}h#ypqRDSBPZ6KU)EN$lb{#x%vuO*_|llwNmU<@`vNybwZ=sr zMe$8S8_;~RU5EB)AwiRiY*_b!<=u*Q)0;-E7d#AS;koluJGu&$EZt2^JELzimDQfH zHv$9xG?D%&{++s=^Ls2PQ_|UrV6N7%*w#dA4hlwr(a|eamv1n0Ok| zfg1wQ!$z+gRjE$vC?4Kns~&phJ0@Z|D$fP6)f^wZSDO2|`CbS0KtViNs6E~kp!9GB zY^A3f&2NPnmW}(Z*Erg1k!eXjn?Q*bB~emEdHHE&eV7P|JDX z2N|!Pg-_o~ho`{%l=I9boZxv)>@p`dn;@1zGW|;BzxSc&m5F3Y>shIHV-aa8NMoa$ zLt#;IKz0sJ_7CAT8$vhv;X5eRsX2Fzy$-gVCBF6S26yHVM-poF7UY0fB*T0`(&yol zHH^=9v{ubMRHAymUI_3ziwJ3X4;uS7sOJ4=e!f0oY1i61rfPDy#q&m^EhqQ8|vK+-v7yx z!LPneGyANgejy}v3SK=J;f1*>jqNtq>F4s@tl^qK%axMBx3F3hUyaeP#M=waDyKS6 z3P{fRcz?n7Ptjki<4_djCVg8qz(gp1xxZ2SX;Zhi=U^p6QiAjPLHsl}l+oUaYBH3e zgaX9S0THyo40uo|LlQlW5KFdG;VUSZbDDAeDsWp*i(`HAL2vgqp%P!(7nNfNy^NU} z+?{O`N*7-fV^6v8TamzdtTOB-GW3BgySB@HwI(L?pM=36m?h<(p zr9Q<$T|k=!ZuEuIw$>zQR8FH8cYe74Rykh+uziGMqf6L<`s$U%&YF0+#GmosZ>d&8 zU&y{7^Vu|8)rd?59-6hsL`LQ90~h87m0$BkaeBh`vEE2ta#AwI6i|mo3;%QDo?|lf zyqRB_bf2;%?v;zo6&lvjGq7a`(L`qE!DMeKn|`BYh|-5j*)Bx*ruE~$w=Jpcr?nqz zFLmJw$}n`A`a9N*+m4KG=DjEOrNcd1fbiQ|%b$(AR`Em%@@hHC>Dn{?I&euAA$Qd5 z{ZV?O5_cp6H?XSU_irKQN{r+oDs9R6IuQriZH4$oqU>LXz2k(lf^DLfj0i+{^_v;I z&r;!X7EuNk;n}7NZ%>2H7v4KjS#^hh9h75j^Jb{n3832xC(0-n{C};_i@E~WWfIQbbnCnh1cv1pV&bH*+v5H*Iw?i z*Gs(&1z{8^CGq5w=0(Kx>Ip{SldU$9SH947=luc%55EbB`qh+vE^Cwm zzXfLNPyIE)fO)9#Zfm2m>!*)v#Ky`~!bk*}DLeV*Vf{niKsKDYpXjH}cLV%?raX?V literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/ygrene/index.html b/content/ko/case-studies/ygrene/index.html new file mode 100644 index 0000000000..6f15b7fe9b --- /dev/null +++ b/content/ko/case-studies/ygrene/index.html @@ -0,0 +1,111 @@ +--- +title: Ygrene Case Study + +linkTitle: Ygrene +case_study_styles: true +cid: caseStudies +css: /css/style_case_studies.css +logo: ygrene_featured_logo.png +featured: true +weight: 48 +quote: > + We had to change some practices and code, and the way things were built, but we were able to get our main systems onto Kubernetes in a month or so, and then into production within two months. That’s very fast for a finance company. +--- + +
+

CASE STUDY:
Ygrene: Using Cloud Native to Bring Security and Scalability to the Finance Industry + +

+ +
+ +
+ Company  Ygrene     Location  Petaluma, Calif.     Industry  Clean energy financing +
+ +
+
+
+
+

Challenge

+ A PACE (Property Assessed Clean Energy) financing company, Ygrene has funded more than $1 billion in loans since 2010. In order to approve and process those loans, "We have lots of data sources that are being aggregated, and we also have lots of systems that need to churn on that data," says Ygrene Development Manager Austin Adams. The company was utilizing massive servers, and "we just reached the limit of being able to scale them vertically. We had a really unstable system that became overwhelmed with requests just for doing background data processing in real time. The performance the users saw was very poor. We needed a solution that wouldn’t require us to make huge refactors to the code base." As a finance company, Ygrene also needed to ensure that they were shipping their applications securely. + +
+ +

Solution

+ Moving from an Engine Yard platform and Amazon Elastic Beanstalk, the Ygrene team embraced cloud native technologies and practices: Kubernetes to help scale out vertically and distribute workloads, Notary to put in build-time controls and get trust on the Docker images being used with third-party dependencies, and Fluentd for "observing every part of our stack," all running on Amazon EC2 Spot. + +
+ +
+ +

Impact

+ Before, deployments typically took three to four hours, and two or three months’ worth of work would be deployed at low-traffic times every week or two weeks. Now, they take five minutes for Kubernetes, and an hour for the overall deploy with smoke testing. And "we’re able to deploy three or four times a week, with just one week’s or two days’ worth of work," Adams says. "We’re deploying during the work week, in the daytime and without any downtime. We had to ask for business approval to take the systems down, even in the middle of the night, because people could be doing loans. Now we can deploy, ship code, and migrate databases, all without taking the system down. The company gets new features without worrying that some business will be lost or delayed." Additionally, by using the kops project, Ygrene can now run its Kubernetes clusters with AWS EC2 Spot, at a tenth of the previous cost. These cloud native technologies have "changed the game for scalability, observability, and security—we’re adding new data sources that are very secure," says Adams. "Without Kubernetes, Notary, and Fluentd, we couldn’t tell our investors and team members that we knew what was going on." + + +
+ +
+
+
+
+"CNCF projects are helping Ygrene determine the security and observability standards for the entire PACE industry. We’re an emerging finance industry, and without these projects, especially Kubernetes, we couldn’t be the industry leader that we are today."

— Austin Adams, Development Manager, Ygrene Energy Fund + +
+
+ +
+

In less than a decade, Ygrene has funded more than $1 billion in loans for renewable energy projects.

A PACE (Property Assessed Clean Energy) financing company, "We take the equity in a home or a commercial building, and use it to finance property improvements for anything that saves electricity, produces electricity, saves water, or reduces carbon emissions," says Development Manager Austin Adams.

+In order to approve those loans, the company processes an enormous amount of underwriting data. "We have tons of different points that we have to validate about the property, about the company, or about the person," Adams says. "So we have lots of data sources that are being aggregated, and we also have lots of systems that need to churn on that data in real time."

+By 2017, deployments and scalability had become pain points. The company was utilizing massive servers, and "we just reached the limit of being able to scale them vertically," he says. Migrating to AWS Elastic Beanstalk didn’t solve the problem: "The Scala services needed a lot of data from the main Ruby on Rails services and from different vendors, so they were asking for information from our Ruby services at a rate that those services couldn’t handle. We had lots of configuration misses with Elastic Beanstalk as well. It just came to a head, and we realized we had a really unstable system." + +
+ +
+
+ "CNCF has been an amazing incubator for so many projects. Now we look at its webpage regularly to find out if there are any new, awesome, high-quality projects we can implement into our stack. It’s actually become a hub for us for knowing what software we need to be looking at to make our systems more secure or more scalable."

— Austin Adams, Development Manager, Ygrene Energy Fund +
+
+
+
+ +Adams along with the rest of the team set out to find a solution that would be transformational, but "wouldn’t require us to make huge refactors to the code base," he says. And as a finance company, Ygrene needed security as much as scalability. They found the answer by embracing cloud native technologies: Kubernetes to help scale out vertically and distribute workloads, Notary to achieve reliable security at every level, and Fluentd for observability. "Kubernetes was where the community was going, and we wanted to be future proof," says Adams.

+With Kubernetes, the team was able to quickly containerize the Ygrene application with Docker. "We had to change some practices and code, and the way things were built," Adams says, "but we were able to get our main systems onto Kubernetes in a month or so, and then into production within two months. That’s very fast for a finance company."

+How? Cloud native has "changed the game for scalability, observability, and security—we’re adding new data sources that are very secure," says Adams. "Without Kubernetes, Notary, and Fluentd, we couldn’t tell our investors and team members that we knew what was going on."

+Notary, in particular, "has been a godsend," says Adams. "We need to know that our attack surface on third-party dependencies is low, or at least managed. We use it as a trust system and we also use it as a separation, so production images are signed by Notary, but some development images we don’t sign. That is to ensure that they can’t get into the production cluster. We’ve been using it in the test cluster to feel more secure about our builds." + + + +
+
+
+
+"We had to change some practices and code, and the way things were built," Adams says, "but we were able to get our main systems onto Kubernetes in a month or so, and then into production within two months. That’s very fast for a finance company." +
+
+ +
+
+ By using the kops project, Ygrene was able to move from Elastic Beanstalk to running its Kubernetes clusters on AWS EC2 Spot, at a tenth of the previous cost. "In order to scale before, we would need to up our instance sizes, incurring high cost for low value," says Adams. "Now with Kubernetes and kops, we are able to scale horizontally on Spot with multiple instance groups."

+That also helped them mitigate the risk that comes with running in the public cloud. "We figured out, essentially, that if we’re able to select instance classes using EC2 Spot that had an extremely low likelihood of interruption and zero history of interruption, and we’re willing to pay a price high enough, that we could virtually get the same guarantee using Kubernetes because we have enough nodes," says Software Engineer Zach Arnold, who led the migration to Kubernetes. "Now that we’ve re-architected these pieces of the application to not live on the same server, we can push out to many different servers and have a more stable deployment."

+As a result, the team can now ship code any time of day. "That was risky because it could bring down your whole loan management software with it," says Arnold. "But we now can deploy safely and securely during the day." + + +
+ +
+
+ "In order to scale before, we would need to up our instance sizes, incurring high cost for low value," says Adams. "Now with Kubernetes and kops, we are able to scale horizontally on Spot with multiple instance groups." +
+
+ +
+ Before, deployments typically took three to four hours, and two or three months’ worth of work would be deployed at low-traffic times every week or two weeks. Now, they take five minutes for Kubernetes, and an hour for an overall deploy with smoke testing. And "we’re able to deploy three or four times a week, with just one week’s or two days’ worth of work," Adams says. "We’re deploying during the work week, in the daytime and without any downtime. We had to ask for business approval to take the systems down for 30 minutes to an hour, even in the middle of the night, because people could be doing loans. Now we can deploy, ship code, and migrate databases, all without taking the system down. The company gets new features without worrying that some business will be lost or delayed."

+Cloud native also affected how Ygrene’s 50+ developers and contractors work. Adams and Arnold spent considerable time "teaching people to think distributed out of the box," says Arnold. "We ended up picking what we call the Four S’s of Shipping: safely, securely, stably, and speedily." (For more on the security piece of it, see their article on their "continuous hacking" strategy.) As for the engineers, says Adams, "they have been able to advance as their software has advanced. I think that at the end of the day, the developers feel better about what they’re doing, and they also feel more connected to the modern software development community."

+Looking ahead, Adams is excited to explore more CNCF projects, including SPIFFE and SPIRE. "CNCF has been an amazing incubator for so many projects," he says. "Now we look at its webpage regularly to find out if there are any new, awesome, high-quality projects we can implement into our stack. It’s actually become a hub for us for knowing what software we need to be looking at to make our systems more secure or more scalable." + + + +
+ +
diff --git a/content/ko/case-studies/ygrene/ygrene_featured_logo.png b/content/ko/case-studies/ygrene/ygrene_featured_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..d0d69114784c8d418e8c0958229d301b3fe00505 GIT binary patch literal 11569 zcmbt)bx@pLlP}KT9-QD3Y;bqi;2vOr0fyiVF2P*_!QDL&f(LgC0RjYfcXwOz?)UAz zf9!76tvgjS&+~NkuTP)uo-^G&9igTwhk;6h3IhX!p&&1<@p_$keSZSLy?$P(b%I_m z#84SMsHTG@)E(puhLNyvFauL6*nzCT8eouxr_&Hv7zPHO5u&9B)l*RxG@di+MPL%s+ymlxAQnh`k8)NlOPcCB#C6T9-!!sNyIEwuZ=iIfFI5RJF{# zKAHYFD3VR5?3a|r1L6jbLw)QT99wOBL$Q69O|4U}4ru>Hm`cZ`XKb6u`QKOV{ za0XNIu<^2*b8rJG`2^TFc!AtJ94wTaKn^~3AP+l`mlen%2;>*!c5i~4mnXA4U~ z4Qbhbt@ZjOLTwF&ItsG0ySuxyxpT2OI9suE2nYzU13B3_IayyhSY15rp&$=ddl#C2 zOOOV;m^(upp%4dq%D)mpW)7}U5$ab>|8om=jw&kuCD`8OUxs=$8M_C_k)4AL$Zlu% zcVGW-yFfL-|3{7gmfJ4EGJM#fM1%IQ$T`;`+vj=N;!i;PzPr%2M61Kdr!^U0qWpl?chi$CH0TDC>d2i z<`Db8M5ez6{+GUBXNVivLe|;Aj`AN12}1rGEr0^D-24)<(gGZ^z`t5Z$N(ky`6Q%) z0#ZCu(o&q%|H4}Qcf0>jtjzz2Wq(zM{jYofFL(c&^cq8dZ~rTJUmyM}{K59GG4K2u z+{5NM_%JX`ISSGeS|0O_J*u3-&={w{ScxPlfOGH{>fWCo$Uk30nbn*6kOcbK%pa!Z`bWo$D@>BlnzA5!$y&$!lea-$YZ0%|3gv>NBBz$mZbfM zgikFI_)2=^`%8M|`%L*t`rn`lZ!15;KMJdMf3pqW3IzRN!oK6hE{h$H1uWs_W$@q9 zu%{+|lUJaTu`DeN+S1C?5CjZt@>g|^r#34il77RJwEolcjpF@jhR%zd!!G}5lU^(= zB0ow@k7*|`_J!aXORcAG3^%q9g*f7)Ya|NNG{HMaNc2y7N1lD&;BE>fMjH$nc0nIQ zvSk%+!qxat5H4m92b0V$BQ^X07#~Cl@F(o&dYNhE;RMuNs?r z8Z0ybASmF_Pk91DmNvyogqk{rp7QuSQZ0kToe|i+;L^wV6dH&sGO@?%$W^abnF|f4lWh2$e4#!(j z#G!?&DQANkH(E^{{N^YQU)Y_4@rhydRzVUn!8XlY zOsrk&q#PiKn+H^1?1IroOM03gMu_l075Mi1{P-CB*Ef$rPIm4z>CXE5c0#PU>%XFkwHx zfy;_q)V+)>J2jdrdwIzRD)UX%KTmIeVCa$u2%TTLr%vgvrM7l3V}s{OJ~;5Sti(CD zM^<(aCDtOX=E=9e$9KjJ1&JUOcLO1hGhNnKd;K{f1uCh*!{f%IeBw9qosv0>4)lhcnj{SpidyW|`W=vo0x93^ zmBxougLk*51&!^`DkYx0(hN5NlRiz``?gOiY4fB6S~cRISw@?H(K)C7xddOZS9HPo z(Jk|5;Y)2U%4-8QbJkNWDXgenj5GCBm9DL&+NyQdDZ_)dOz}mDZ=`%W>oaU)&FQ+0 zGX3QPdtHhCoQ$J~(Qu}557%Ye{Sd;gA~tDF>z#NU-XhIHptgFi=K!7snnqFA@fXJ@ zctz*jnR-%kQmIn^gu7tuDl?^48=xPuFN%udbln`)3|o zsLcjn6Grf{-^~1(N=!-_%MePTbPT-+^g^g2Nq^$Q<;xds7gh(vt~NFdkS~uSR3W(h zWRT;hZ=gmxNM(kVwGhy`P446pKer zKxL9vLGPJASMl}IjyZY!a7fGRa+;orjyu6LDq4663?tukyXYbt>PNVPsot~2^95g%z*BRubO z=7bNP;$3X!08^7GhS6l;Lyc}+kU@KyK7wWHMwj?VsZR_vJaHH<=RF-Oog;?cs`#cP zqrnF1p*u1rrxDRQE_wk1YA?d~!T|!i&tDKB6v*m>FTz(X4Gp}IcleU6oL*Cd$P`U) zD_!;%q{7k{Pm=VrW$@mm5D@-WHUv*fNx2v&;m*A|n&!O`BujxG3)$;;tjlJ{nJM4g zve^R`Rcg_YnyVRuCX7g<|I{P8ao0Tsfu6A2P^0A)nsyNvPMDpa2@i~7L>{hrd3Ye) z>50(Wv0w z=p0=X1)#Vgw>v#_z*-NNlC_!TtBdHCq@o_1hkAixHpc9pKdjT^!lkqXB0sHh*~|^9 zhL6{RF9=sLi&q`dgG%p`!D1X4qkH5gwO*j8hcD~VktZBkv4!jfcmN>-;g3|^*3eJl zTx_U(ghbD1yR3e6X#QA%)?{Jwa9DDrhKnDWGbltBJQ^7;V*=l?j@>KO+@-Lf1)#Am z_SJicDiRlqZ?>54sJ4t|zF-uZ6}}Vzp{U`1$1K;KMFMvZ~?V;7G=RcL`0~-mIk1TX5I? z8p!#@SM*z7fVk;>G$bu8ZE0$;G;?8pg?B-JK~4|;C6uz^!7AXPh6nB7x@BkrDRuh?{XI5(_WqRNaPi9n422xQe^N>WN@ zDP}CsNM|mUOvV_mM7z5AB#a9Avxk-iYc|QufZ}8yy+4=`%|OX*DI^%q6tBE)^Zs-e z@qtUcs3^Il(^S$CFYgHDF6Bn&tIj)orVX>4G4~CO87l)8ChkH=6QOCkbdE++_+~4<<0m|rYsQ*+wtkLRK3=6G+HGe zhVkp4&_93v@RrmzmoX8cYU1mwshN%+xC+{VKzRnEg!WpaC=>NHO>)#fr6;-Z&=I#d z)3n}&G=zq}DY5F^=|*H~+3XHMV@p|>*e@+B3yB(tCFb&bY)YJ?_E0l7$J^b83W^vz zXliQGlI0a@71K>xb6WEl1Wl-Y&dH&l>k;KV_J*;1z;aPbz7LGzY5@?yscE|Io*J>5N4v&pw ziFp5-fTsaXS-Opce%%9pORro#E78wdwiVVc~d?lP~Pp`_3^@|vvnCF2EUHC z*eEtp`>^D2-2|vbIu%STEa_a{I{NyM;r>9wnd+CtjVsvbeBl9^T9@xTqgiOfjSRJ| zCP%ZdJaE08LE>-66@%G@abyV>_0&F#Idun@X_i1sQYy;J+n=`wL-sFJQ$|j(dJVK~ z9UVtZbN4TQNCYVsD`yKst!0L5=5rX)buSj?=^o?Bnb^Ns`;mbSK9YaxyNy6)A;;z* zHkvy$&T?oKoAD4LZT!AATeIQU!7d;~12u+r#U;Oa^Clesc5dgu{df`4#4NCvMClRz zZ7ape(3g@95fk65Bwbeh@8P`|uC7778)d?5p}zH*00Q1R~ghu zAH^8uH8owp;H^Rhyc9P5O09Qanm(6R-ZvB%hnjQ-Yyf#-B0>=F3PLtr_D9%#pUv$wkzaA;_0*Y%?MSmmlvW3G01co_F6ihX7hXp{q`%F23-KRJXeyt$ zvTt~9P(H%oK8U?q**TaQNayVOv9+@^mNP(uRJxVrz8pZs%vPb0Hkn*k-43$eE{GGO zmtP+aL32BvN8AdRh_l}4JdWbU^n=T8HLlj9d<-rj6LGVh{gwxV2}m;byCc~3-;9uk zjdbdtu>W*)Qmva)jigtY=^D*MH?g@<%w`^SUiqff2X;Ljqj8Y1_HBrC?KxV2$ zdO|Vat3r?^q9X7!&WYD=a*@8BWH_`JP1wlADT+v}IydKXGvZN9HP5HKLtDkaATsP& z)X_A&nao)R4-MY3F&@HZh3?6su+aeDjs7xImQ1n|HP>Oc5HJ{2D)Hv-2_;*;qP0)_ z()|lo74Fi!p4V;rGpi903kCsb&;go!h7ZeaOQ^O$#L*G(S6CMluMN=t&~p7{Owt6!NRzj4#Gvz}ZrK;DP6Z^r*JZ&pY zauOcD%4*iX$&Ko&mZQ9{Hg?Tty_&bhZZdh4-$C!&=Akk~e=m`#)vDI&q}}!4k3=J# zQ9tS*RIe#(Zp-f3pMx;j@Q0B#0G|%Op3sY(F?X>JcssU~p)T3#yVW2y{!ExIy%e;q zi*$anSVH65sJ7|&Q$P((Jqbo>VpPg zM^Tvh&@r>GYYe7vegsd+iLjizST!ThzR0-!;*{8rJzSqE%f71U@ynsXbzzidB|%{8 zVO>ru5w1F8kdF1*yi~mEAAV9t#l$Iv*L8&f*gdPB*|N-h>=s7YUPMpZgDZKmQ|J0= zSqNLFw$LB?2n~;TTNKL_>0x#Ua+X~g>fjK%9a_)c`%`jz*&0OxLrfS{Gl6Ek;fIwh z%|=Kfuull8YlMa!=tyuH0h!V^hxU`7r5VZKOukiB9VbZi-r7br)fx6uN*l11XujK; z__iH{WoYXL;9p09ha->b6ZXD7%Zj#Z(W|!fSVN(zP1fVS?LaRs3cGOoB@uKywydIo}?X!Yry^g1O zEi0{UvQ*4ue1lM*QITs-sh)v-4Lfs{s--@Xz)Zg6^mAp+`wy@^IJ)0ktme%mae=pOo^^Sv7fZn`-n zS3G>@vYe?ov?bU^u4BGNitH0g4<9}%7!3ZE?fIIPBDBxfXtJG*Pnu=B&Aqn|4amT? z2tvZ}+U-KlzF#`=mJUX#gwI0I>)1}G(=1U&nw6-Il8I2IU_^;V(=ja28Gyo{6LDGq z5XfYMsI<*-&HJN`{T{x)cx;T`GEv_yTh_i6yu7M|X!h;9786Z*^_?#it7N6G4s}(F z7v*l^bv(xq2@bq4kwkR*voH=4eh7K}THzP$rNXI=R#sNZQFtJaWue?zG zfHfn8IhiNN1oTgat@*AJ4wv~PM}0F2KN=$O3P&W+atif=MC=Wz-@U~gxi|FuGf8v4 zLY#hKlePtc+mMp1nEqitplbSzutSoLLbA2&eT+jh@LKllo><%vTjSodsBB%FI}@@Q zjUOC#09FW?liBrK8Tt4KFQY}2eu80WC`6rUDU>B?tBihFI4J`9Eq)h9sMRCzVfr^& zjTsZArOd}E31HxbOz`;LU;g^-n!D4)VNN|E?^Ao4!_nGYBr}CKcz!rnI~3|Y_dP>X z5=am9^B>%p-mb+!?fc=Hb9LOAMvzi zy01LdGFvLY2i|aqABs6*dqVNeH8Y7~z?rUt1Nm zjg6P1_*J9o!ps$7}AvSESOwJMucAWJ%ms* zWq8{Zzf=F4XokGcolZF`4E8x9zT3*(gUCspM?vYF9~2}&?bML`w-jFcmub&eXargj z2n?NMa>RxvDwN%+&$|Wn{HnbJtsKZ(^B=1?NHA)O9qdk0F}cDF>=Yvv?I~BM%m-ro z9>kZLl~(8VA;n%I6z@M)a6Q8hJK2P4R@%RU;<%S}&QB!eDF* z)!dw@6UDG8zx>uPi^)Yu8QHq>zzuyfmEYbDdblR<ZhQI0UOfE9#SL(O{FG6tlT_%ifb3d$r_mR9}gC{_49)6x#wY%>S0OP+6n>qT5M z`^skX^P$Ln^R~PoRZb3RewS@@M~iy~=hgzr!t|<5DGmjO2ptZr93+OLJI=~4j9;+i z4zK@&1Mkvuulu41nVFeeoD=W3bZZl`xHuGmpgj7rp_?JiE7k=zKK<1SH9`yRcm zKAn5|o<;D%m`T?6H3uKl=HJ-{N~Zm6`p{1|{>QnQ<;enxXls*c)&5;_WgHMo^+S^$ zjVc#?5K2E%Fx*lNOj*ugWHw#SC)Ho1;aq8LYKDj3#XDCz{HS8%-;A26O3GOyk9yHQ zv7cce_@rTvMi7oqSl?`X&z5DNJ^-?0Y{Q8w-zOS-?O7n0nR4_MLWj*lh|i7LEO8tz zksq_ppw-iFWC05ujKNVnksJtB7OA1mf0&M4B63HNM zi$whZw8<69pHY4m-Qw@~z~T)IcM^!T?dmtxi_R<(BRDI4UOVhnU_9 z(04(ktJ-%>)2lB!ps+`xk4`R*j9}-`9Ek%THLJQRlz=;WMyMg9IOR6aBg5y@B;#r?iX%OR!$uG?=3)%W4Kyl{iGQTnf{DW*PmM9syaU?i_vv$#e~wI(Vh_cjK{Swg{y{RJ}} zb_(DJ75XpX-LdLu+J?IB=bL*GcmwWk1xxlXyBqI+vn)Td!S!L18%)g366G|79t*o4 zlkqDor7gJ&W{Kv+nVe7l`jm<=dEhEpIef)KpeT(DKX0ceBNH@DBF4Gq;_C~4J13ac zKx)(S=qL849Up$9EOBTM_}W+SlcN&;m7AHT9({=rX95*=P#dKMj&|m%l}y872bix1 zfuC6f%b=skCsG%xsX(f^0%TO5y`rRX<0TXQSxx*{axTk*KkOSDC{#}$5p}8VyWY6a#M-1-hO;oF@g9~C8;OqT5EoaMaA2`3VcqPMlB3eZSY5~1SQI?G>HBa!@IHUg;&<`( z_RWfk%jO_XQ>W>84(s~jsj?dFnx{fL<@1a4>h}9#3{u|m z4INp@gXYC!EYZi^+X%eAzP{vWbTz|8!`F5i0{*UJUBAOU4#TnKXlp)U!>v%B8a0bO zBe#g!(XBb7fuY@lul_a{{KfX+;$o(p-}Csz*bA{^aw`!_zuOr-kx%v8r`>c3Iy&~R zj~~VQ7{}>mSUxzf_bTLqh%nE(@NPX{3ojAx23*mokAb%;3_bvA1vMm_!imz?3gZ45 z$t062h}{@^)xh4!5N*Rj{@MvapL04SEzq6apMP4 z4q#dO&axE4BG-uM#@qDV*|+CVX#Gy;`3aGgIEil*27YtMf4w+ zz69hn?pSTWzT-rOQr*U`+em7p21U)hxyKa_ll%1SlBLQ(EIwgV|6cU2 zNsB(HCjA?g((Ha`1$jhp*J>np8VrvbY5DbO5Ecn~TTb_FtRJqi?LA%3z1{KHS33vX zCQeP4pv0g1>N;-(={=n#5Bs=BVC7vBg%nSO=Y8KsBL}Zq@Yv5)^K037 z8mJOL7e3=4-rb3IV2MI^Ns~<)3#W@mU?oP^-rnkIYwzrT)rnicSXM|Z25Xc_mT_i8 z(*-HQRC3709MBS1=~M*s#vsA@O|17ac!Zeqd42Yw5c6AH?Fza%TxxMWNLjt^y9jG4 z%G{u>iH>fXn$DdGc;9T=<@04IF>zs|q~l9{AWYzq4LGHVj+?6O9j@Vb6`1#zd0`a( zYLJWMudJ-2DajXr1Yt|*2lz}8uc=Z*wCy4j2O(_6O(9%etRcTMqY=F0RleiZHMHxJ zC{Dk;!r>n&KJhE`@V>=slqu@P_PxE{x}s-Chv#Aq z4R9Pe1<@?p6B4@v)VLK%F+#I{#RNLIPbZ-vr(jf-wK?8j5<6D;_=zFLR2 z2UoZ7nrlZ!qJFgb_&!Gz4SMV>z!4od5JB2Tm5f}!g(~Bw^Tt|RM~C;dEtiv-Hf41p z7t0xI^Ke4G9FrQHpmY)WvjfOz_2{^ajpqMwcyVP4H5&rO$K)#cUCn;F7-{sFznJ6> z+Bmab4R%S=YRkUI`|fWH5Ey&y3(3}|as^P{wJ8n#t3;I7@6E=LQ>9N+k{Xx z#P#^LsPZxYsTpwOc+2Z8#h_fSFL1vlf~qfw2Ur(<+MG1yC+k?impUNRxi5rFWmlIAX8k!udS+}ts+Zy@1Zc?K_>M0-! z6B&2&dhP^sPd#oDvvIVdu?=?t;Ygh$>UHb<2x- z_Oq(z+YBY^>L=QUf3bU|I!qW>2H_2;|78yc{S-bi3qr<2#Ew7Rc z?mR&Fl2WUta-rzNtqv~%!4Y`Cw*&7p85tROZ6Trv6Ao=ps*;hgs4xrMKDR|Pl3o5~ z5Z{2c1(KBbv`6DTOyo@`%$MZ*23_XwCZPu0-a0k5jJhwAx5e&A?lbl6)U8<9nnR)Z z8E@m{a80|npXw&=8WeeF-wNnj_(@l%rk$Qg>8$ksXwWUPKnh?+1`EsJ^z)fvr8plXhB62v%Z^n?H z9l`N;x@T5!NYsQC>2HZGJe8SqC@e{A40V$Yv!mEy9rdRlDYHaM+ojnp zQHt|1=&J2TXTi2181i5W{AN8-j}%nZmf_oF{h@g)4i&=G;ZCN@5aZn;X0(Nt1Bd^+ zGR)Ek1>{aUA80Q9Iz`}56H?%5fl|PVU#2-YJi?jda^I-S^K>)Yt@8TkXc(7!l>5== z!xRgPWtjm@%$0lFrK2;dT!Yt+;sc}x5_PH zFGLF3cKU<=bngS`hfH@3$`nPhozBhz!Q7YJ{56om#tvYj|W#)okvX(xz}~ zc5-@zwdLa_?^}Ovr!9GaS!^@8aY}be@#i=21St0#3OwETTE>h9XkXv2A*CzC=QB7F zz@U|XahCP(1alv4VpaH;U`NzGzfO6O%_v4F0|^oP!-#oP$Fl`~7*-fiz=y*75NW&o z*vMZ?Ra7M?JjbBzKBy?WMxsiH5gL(k9gbKYS5J3p$Z=0HLU=JFCBl-u%8ZJ&x`@Gg z{-!p&%>2XX=@ZyG*+Xg|D1N`8lkL+kOV1DTWtGQ+q`Q<0#^A}_*}`ZFv7RLJCJ`lX zt`Qk|BN~IZ3AD#;JI6`PFJA5Ul84`j)cu+6=-w@~2cAg(jF1b25ywj)@#|Qz;1+#6 zl{$-h@z{}-qh4ZneW>6u5k$SSw zfKpxI=Y9pXauhpso>24%g93|-7f_)YtEGC2WsCQi^ckvG3<45Syms4kUmK7==M=-0 zdM{RKkAx;+ifif|C8t>I0a!>B-A|`u_RG26ZrWqqipfs*5`WM)ix}3vV2@$_dr|6N6G2+=GU6psdB6IPW z3Q^YsIy)IQcc;gF<*fk$Bt0&L2uphefo4XLDP4}}8>4@g_3U2YZ+E5l9V77^5)&-f z^sEoVqPlM<&|kREQLqEYqr=u(+a0H69%|QV)gPe5Gs}XhOBV`21Nc0$rUc8Y^;9lQ zai7(Ihc3d!$37h=64B%qoUc&lZA2gG=Um6gA6v;e%4x!LrsE826q4O)v3`g$M1mNC zbd?Dt_?CxkFVDg~SqgCgB^1r}v*llc@IkAoVsso%mTO zF`z|(FLV#76Scz#R_WK(H}scs-X=99S5Ols>0o^%hf4OeqmbU1&dYV;IP3-TBAyUZ zs}d|8N`TgIKE77osr{&?Y1!-GgQBN^L@AzO@R&%5O8Ffj`B=}O0wUBJPc9f@@Ay1f znjHiB{z8QB2F=+5r;YLQub_sG1ym^EdI1h7EkOKldv2XE-1n2X^mt)X=vvxV0hlCO z8fu?EZFM8My1GsivT^h9hDd@;9)8(>6CZ-w+U3T7j7YefEX1#CcQ8c=#$E8}*cLec#@dwMtz+3K+f5(rlP;CSCAR zJL&=N^yD1fozCM2!ikDyzzimbX7csDHR|pLwk1!q4j-t3K+7-wOue;{@o=OWk5=AG wLADg-pP|G5$2r9Rj}gXyLmyYSUQmEAkl$A5`U*}qe}|YAWK^ZABuxYV0}_K~sQ>@~ literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/zalando/index.html b/content/ko/case-studies/zalando/index.html new file mode 100644 index 0000000000..49bad6ff9e --- /dev/null +++ b/content/ko/case-studies/zalando/index.html @@ -0,0 +1,101 @@ +--- +title: Zalando Case Study + +case_study_styles: true +cid: caseStudies +css: /css/style_zalando.css +--- + +
+

CASE STUDY:
Europe’s Leading Online Fashion Platform Gets Radical with Cloud Native +

+ +
+ +
+ Company  Zalando     Location  Berlin, Germany     Industry  Online Fashion +
+ +
+
+
+
+

Challenge

+ Zalando, Europe’s leading online fashion platform, has experienced exponential growth since it was founded in 2008. In 2015, with plans to further expand its original e-commerce site to include new services and products, Zalando embarked on a radical transformation resulting in autonomous self-organizing teams. This change requires an infrastructure that could scale with the growth of the engineering organization. Zalando’s technology department began rewriting its applications to be cloud-ready and started moving its infrastructure from on-premise data centers to the cloud. While orchestration wasn’t immediately considered, as teams migrated to Amazon Web Services (AWS): "We saw the pain teams were having with infrastructure and Cloud Formation on AWS," says Henning Jacobs, Head of Developer Productivity. "There’s still too much operational overhead for the teams and compliance. " To provide better support, cluster management was brought into play. + +
+ +
+

Solution

+ The company now runs its Docker containers on AWS using Kubernetes orchestration. +
+

Impact

+ With the old infrastructure "it was difficult to properly embrace new technologies, and DevOps teams were considered to be a bottleneck," says Jacobs. "Now, with this cloud infrastructure, they have this packaging format, which can contain anything that runs on the Linux kernel. This makes a lot of people pretty happy. The engineers love autonomy." +
+
+
+
+
+ "We envision all Zalando delivery teams running their containerized applications on a state-of-the-art, reliable and scalable cluster infrastructure provided by Kubernetes."

- Henning Jacobs, Head of Developer Productivity at Zalando +
+
+
+
+

When Henning Jacobs arrived at Zalando in 2010, the company was just two years old with 180 employees running an online store for European shoppers to buy fashion items.

+ "It started as a PHP e-commerce site which was easy to get started with, but was not scaling with the business' needs" says Jacobs, Head of Developer Productivity at Zalando.

+ At that time, the company began expanding beyond its German origins into other European markets. Fast-forward to today and Zalando now has more than 14,000 employees, 3.6 billion Euro in revenue for 2016 and operates across 15 countries. "With growth in all dimensions, and constant scaling, it has been a once-in-a-lifetime experience," he says.

+ Not to mention a unique opportunity for an infrastructure specialist like Jacobs. Just after he joined, the company began rewriting all their applications in-house. "That was generally our strategy," he says. "For example, we started with our own logistics warehouses but at first you don’t know how to do logistics software, so you have some vendor software. And then we replaced it with our own because with off-the-shelf software you’re not competitive. You need to optimize these processes based on your specific business needs."

+ In parallel to rewriting their applications, Zalando had set a goal of expanding beyond basic e-commerce to a platform offering multi-tenancy, a dramatic increase in assortments and styles, same-day delivery and even your own personal online stylist.

+ The need to scale ultimately led the company on a cloud-native journey. As did its embrace of a microservices-based software architecture that gives engineering teams more autonomy and ownership of projects. "This move to the cloud was necessary because in the data center you couldn’t have autonomous teams. You have the same infrastructure and it was very homogeneous, so you could only run your Java or Python app," Jacobs says. + +
+
+
+
+ "This move to the cloud was necessary because in the data center you couldn’t have autonomous teams. You have the same infrastructure and it was very homogeneous, so you could only run your Java or Python app." + +
+
+
+
+ Zalando began moving its infrastructure from two on-premise data centers to the cloud, requiring the migration of older applications for cloud-readiness. "We decided to have a clean break," says Jacobs. "Our Amazon Web Services infrastructure was set up like so: Every team has its own AWS account, which is completely isolated, meaning there’s no ‘lift and shift.’ You basically have to rewrite your application to make it cloud-ready even down to the persistence layer. We bravely went back to the drawing board and redid everything, first choosing Docker as a common containerization, then building the infrastructure from there."

+ The company decided to hold off on orchestration at the beginning, but as teams were migrated to AWS, "we saw the pain teams were having with infrastructure and cloud formation on AWS," says Jacobs.

+ Zalandos 200+ autonomous engineering teams decide what technologies to use and could operate their own applications using their own AWS accounts. This setup proved to be a compliance challenge. Even with strict rules-of-play and automated compliance checks in place, engineering teams and IT-compliance were overburdened addressing compliance issues. "Violations appear for non-compliant behavior, which we detect when scanning the cloud infrastructure," says Jacobs. "Everything is possible and nothing enforced, so you have to live with violations (and resolve them) instead of preventing the error in the first place. This means overhead for teams—and overhead for compliance and operations. It also takes time to spin up new EC2 instances on AWS, which affects our deployment velocity."

+ The team realized they needed to "leverage the value you get from cluster management," says Jacobs. When they first looked at Platform as a Service (PaaS) options in 2015, the market was fragmented; but "now there seems to be a clear winner. It seemed like a good bet to go with Kubernetes."

+ The transition to Kubernetes started in 2016 during Zalando’s Hack Week where participants deployed their projects to a Kubernetes cluster. From there 60 members of the tech infrastructure department were on-boarded - and then engineering teams were brought on one at a time. "We always start by talking with them and make sure everyone’s expectations are clear," says Jacobs. "Then we conduct some Kubernetes training, which is mostly training for our CI/CD setup, because the user interface for our users is primarily through the CI/CD system. But they have to know fundamental Kubernetes concepts and the API. This is followed by a weekly sync with each team to check their progress. Once they have something in production, we want to see if everything is fine on top of what we can improve." + +
+
+
+
+ Once Zalando began migrating applications to Kubernetes, the results were immediate. "Kubernetes is a cornerstone for our seamless end-to-end developer experience. We are able to ship ideas to production using a single consistent and declarative API," says Jacobs. +
+
+ +
+
+ At the moment, Zalando is running an initial 40 Kubernetes clusters with plans to scale for the foreseeable future. + Once Zalando began migrating applications to Kubernetes, the results were immediate. "Kubernetes is a cornerstone for our seamless end-to-end developer experience. We are able to ship ideas to production using a single consistent and declarative API," says Jacobs. "The self-healing infrastructure provides a frictionless experience with higher-level abstractions built upon low-level best practices. We envision all Zalando delivery teams will run their containerized applications on a state-of-the-art reliable and scalable cluster infrastructure provided by Kubernetes."

+ With the old on-premise infrastructure "it was difficult to properly embrace new technologies, and DevOps teams were considered to be a bottleneck," says Jacobs. "Now, with this cloud infrastructure, they have this packaging format, which can contain anything that runs in the Linux kernel. This makes a lot of people pretty happy. The engineers love the autonomy." + There were a few challenges in Zalando’s Kubernetes implementation. "We are a team of seven people providing clusters to different engineering teams, and our goal is to provide a rock-solid experience for all of them," says Jacobs. "We don’t want pet clusters. We don’t want to have to understand what workload they have; it should just work out of the box. With that in mind, cluster autoscaling is important. There are many different ways of doing cluster management, and this is not part of the core. So we created two components to provision clusters, have a registry for clusters, and to manage the whole cluster life cycle."

+ Jacobs’s team also worked to improve the Kubernetes-AWS integration. "Thus you're very restricted. You need infrastructure to scale each autonomous team’s idea.""

+ Plus, "there are still a lot of best practices missing," says Jacobs. The team, for example, recently solved a pod security policy issue. "There was already a concept in Kubernetes but it wasn’t documented, so it was kind of tricky," he says. The large Kubernetes community was a big help to resolve the issue. To help other companies start down the same path, Jacobs compiled his team’s learnings in a document called Running Kubernetes in Production. + +
+
+ +
+
+ "The Kubernetes API allows us to run applications in a cloud provider-agnostic way, which gives us the freedom to revisit IaaS providers in the coming years... We expect the Kubernetes API to be the global standard for PaaS infrastructure and are excited about the continued journey." +
+
+
+
+ In the end, Kubernetes made it possible for Zalando to introduce and maintain the new products the company envisioned to grow its platform. "The fashion advice product used Scala, and there were struggles to make this possible with our former infrastructure," says Jacobs. "It was a workaround, and that team needed more and more support from the platform team, just because they used different technologies. Now with Kubernetes, it’s autonomous. Whatever the workload is, that team can just go their way, and Kubernetes prevents other bottlenecks."

+ Looking ahead, Jacobs sees Zalando’s new infrastructure as a great enabler for other things the company has in the works, from its new logistics software, to a platform feature connecting brands, to products dreamed up by data scientists. "One vision is if you watch the next James Bond movie and see the suit he’s wearing, you should be able to automatically order it, and have it delivered to you within an hour," says Jacobs. "It’s about connecting the full fashion sphere. This is definitely not possible if you have a bottleneck with everyone running in the same data center and thus very restricted. You need infrastructure to scale each autonomous team’s idea."

+ For other companies considering this technology, Jacobs says he wouldn’t necessarily advise doing it exactly the same way Zalando did. "It’s okay to do so if you’re ready to fail at some things," he says. "You need to set the right expectations. Not everything will work. Rewriting apps and this type of organizational change can be disruptive. The first product we moved was critical. There were a lot of dependencies, and it took longer than expected. Maybe we should have started with something less complicated, less business critical, just to get our toes wet."

+ But once they got to the other side "it was clear for everyone that there’s no big alternative," Jacobs adds. "The Kubernetes API allows us to run applications in a cloud provider-agnostic way, which gives us the freedom to revisit IaaS providers in the coming years. Zalando Technology benefits from migrating to Kubernetes as we are able to leverage our existing knowledge to create an engineering platform offering flexibility and speed to our engineers while significantly reducing the operational overhead. We expect the Kubernetes API to be the global standard for PaaS infrastructure and are excited about the continued journey." + +
+ +
diff --git a/content/ko/case-studies/zalando/zalando_feature_logo.png b/content/ko/case-studies/zalando/zalando_feature_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..ba6251050d15abb523d009ad79df06cc1fa0779e GIT binary patch literal 7643 zcmd6McQl+`*Y{{4?hv9z%NP+Y%3v@`^xj*dg)jz#QO4*|qnGHti(aF5qIaT4ZxI9$ zJ^Cy6{oMES{qd}KJ?s1H-D_QIowN7;?RL&S*E-jA`{i~OKq>8lFarSOF}C9bb8vA2ghgHK zjA2%AG|&WYj<68{?Kic7fCy6&&?|^Cm$IEC+yWuzhJtIlsc6C6tYA=6kmw7bu#2Dz z(hdnn8v|XC);11;E+U{mjSJq5f3rD3z&};cRw5wr-(G<_%4$GKTNE4!;oxJ3fx%!P zKa>Lu5#Zs5JO^@ffgzk+P);xxI~Q0G%q7Up1^gF*?!2K)%>*^1Wd7yrZYBcyS5nT- z&K%A>9JVNPPB0V-<>cb#HZ6hw&e^9kg8S;G7z82U|xJ41TB1@HhCb zcK^Gh-->t92&$qGcZ*_dEoBRHM8a**@=_w8I|+v=!c>qK1{2`p;x%UH;^X6H=Y^W` zvKw;=aNiY>iyO{uVgfdSL;tq(pYS|T0VxQ#gg6hk3MTY^wV8iV%*LH`~VpLJz@Tvu>PTc z!7(u~?xc6mKZ5^uaQg{Bj0f-ne8IvX24E6nU=d^7b^-G5&JL{Kw*P&s;N8c^y@!MS zn}_vV_KySu6ASMi?tN?=A^_kXCI%KJ7WTb+fBY8TNw9J50dVn%AKjkdbZMGO!HEXiPDcz=4sMfu>~|KWj8`}4QC>uIu1 z8o!odi-OiqEH|tj*B(s@hh@qll^=gB7Lt9fD4G-+RzGPJuRw75{rE$HZv37hX)bF= zk8*VIeQzP<`2}^bRNkW@C6$<#shF2u}53Jqf(+FpM=WEW>2nC+K`tmH^B2% z9?0_URs#5eM{In75|m=tZh&9Qj%WN#a~>2-I-6|#Aww@?gM(6*=8H*$7)+J(oVIn6 zf|ACW`kQ2Hp(Fb{E*Jbv7CCcOR!cCQT(;H;;+&D$kS;5#xv`QiD=>A619?t#?$bpx z-4Ka4SmksFISFsxVt0gehYNT=+#;sS*cw$kTv})%_R@PoBdbfa=LI{fpLiT4>tK28J z=u=L|Q!|U+1>tl9%00nX&r)^Duc20@&(QOo<)kiMg7;h%y9~{?QAImN9(lI_*YxmW zjcu~o6lCon_ev?q>ZEAbAsEq6$RvBk{-sBzS{<$;S}B3IvRt9p{%yUR-unT~(nJe3 zK(c6BBSM_8H5cOYK>gBBFhE)N-1!&Qx8-giuF-x=zNp5j`jbfvLt+|BSLa{P(g!z0 zsgI{id>7}9T&&og3rnAao)*mvdNhLYF-NA`sZzV1{G3ntp)P!xWRZ(8h^8WgPstVz zh%@8)!-#nVrI0U&-z_k$(Y4T@TQ1Z^#A*?;q;-2~|0t|Kqo1m)S&6W1i&oknivxZ#fbdRtn615BuKSNwNx^F6fm&F9ov^@Hk+N0ILCMu@qPky`+#IsZp?V#nHXlo9iwpQ#{< z@R0+AnVG2f-Y|)-J-w7z8bFp!Nou6;<CD9yzPNNYZ(X{^VtiKlG|+l5FU08e zy{SF9>g4`6d-8gjQE47u{6^Dg#o!0`6FW4~;N-cc<*G=7(KRp9&(?JeP=dnai+%lT z1GLEa1~Dx>&T2~HT1i^1jJ$F%Ya!}gOO>?ZWHzl^z#}#*8-MruGkh0+Ti#gU;i_-1 zwg$s_iidjBw7*=>7VW?Qg}w*$5pzW0wcHcB77@GAfwp!ls!;pyE$8Jsm#-9!zRwF8 z%;^`b-CI>ju!7)U5ozr$0txj}msQT+<&1>J5BST5|^}b&ATTDyC?ISlOL> z*d+0#$Oep|lKEm7L_DSJ`IJc!r|EOew)iE*0Lf?uQy;RAhwYGc3O_=d{mf zS9*u0W$ie`FmuTs)C)Z|Ajb4$ z2$r0=Z>iU`D)O`I_lt(eJY^cz@#*ur*qYYYulYul05?E7ONkZA%`@_OdyD7jOzO-J z(IWkr?KGa~;PE5}v*sIt;`bG|fRY=VoYT&+;z_r1?M*XCD-9*flK_3w>RZ6Z6ANwA zjkeP^_iIj{*44|Kt92EJqW!raBPpMq-J7<{pbc*Yq`Y_L+HG-O7 z2UNhr9Ucn(0SclFP>)T8b*rlZ#M}u*uyJPu(B9*f6xY?hTYx=`ug~pREISWb*0QVA`^RhWNX!(|0qG_MN5H4-T;XEbGVN<4@}*P zI~JdOQ|*&n)LlP|h`ZsoV~98e1Pm{6fsZyT7FHJI$oNSH1!@!&eP4$AxxgdyB@%Oa zDA!6s=X5LRMQBS?2;wzTaDMglC(3_a1?4_$= z-%TASYMiC7(0V#)L1ArlP-B%=TkV4^;LUi6|HP`ZpxEq32rARe!q*J^Na@RgxyRe` z6t|a$xL0${jyF1mUIA(T5#-=YlG6i(QQ894nyLU~LxUpWCu2;OrM^bWH0WNrzTPPh3}PQdFt4mFndzD)bfdFlgG`g&D|`e^KCJ3s%MRgWX8i-%;`)TvDixudSUCrpBR&y*y@R$wuzqXdBzan*r0A?b%qP~M zkLPqo)Xizptf^Z@nTTkeT5apz&1laMvz09Czj(Q}f7t`SrovC!KcZA#1AdH*?|hQGH2QMV>E_ z+cVqF6c2N@M>zPihewBAB!`iW+47Lcm4d@V{6Y*!L#>C6nd<;H6B`MPc~N9#-QDMb z2f6FF0Ha?U8)lQ4cDDctH9oqQsT~!~(rR|EhjNL2@iJzQvpF^FwG@D~NxTM>LNu8D zl(<s={mSHht)yHwc+B;%+j$+R%aSbc6OE;)Q&mHb^V$*GRCjGl^yv zkzVygnUQ)+4U*+8v)ArgvxV!czuxH;!D>pdOd&g?i=A>;HMEVO^aFUw*n2e-I;VZ*a=P)GqgdgM$%pW>uRQ$-8xX6Gk zG`B(djQQF*pj;`gjI0n& zh%jXA_Aw>+Vpp0)euCoPCyXFhD~dUj_b!~fnHHhj;oYHZQQ4X7A4FdX?9-$RNll;c z#mDzrPf$POk`dBlDe){Vr6})L&DdOSE`Qxfa6xV>*JS(bg>B*H;Ct7rxYbr+((dR6 zD_{0n(fZWboMq+uLh{^J_aTh~hc+YbFT;oKL&J+3o3TSzvcI-sdG@5Kd2RvK?&I|4 zY?_Rbr33uzex~N@LSq~$GZB4B+FBvhzlu7TA!hh_GQKw+qhku)Y`w@^!1swdTmjA8 zU%0&?UbNM*uloFafcp$s;?7YnRwuCR>+PN%e{Crdo zPRVdkc?LwmQ}AQoPAYas4CS}*0fsHH!4LQQT@TdxLdW&;-!JV5TubNFToHbpt_jiV zi2II&7wxWo25f7zc1pdrX zJ09pjEw5(>{Hn@#4DdbE(yw`zocU>o7O9@EOI$Ku%x=Kr7p$Ne^idOqUum=L%o=l~ zi(p)0`F!nL^z9Lb0#GeHMsRi7`4+%0vw90SFKO_(DyAo?fom<@wR>+RPt@<$xmM29EQz%2#!3wUadwve>e zr#_|2wYL;$--hz|Lu_n)6kvFWga9Ju`+oR}*!M?lyfH;`Sfr#pKM^+Q9rV-%AXw1B z#l6*@bFn0FI%x5(d{fo*R^)e9u9`BPN@szUTzVyus4-dX@w_Jzpx~hwJ~8ceI!8Cl z_q;t?65RGCx&p}_`NUYB4};VDb99q4D=h{5KX&a|6;Jf^l1S#8S&rg=C{bv<#KQ@Pk6f-Cfp6h6+jsa73s`*Gsq0BE5HM+;7V zIztt*$$Vc^{X5VmZ8chV^`g!9%7t1ZDf6dofR-Y)G;Ei*jQLYF*ew-`2dYlB|tiRxP17sV!B zty!(%UN0HyvR6j->SPp6hV)KuEnC0HKW!~lf zNpLKRY%Mz(8dF3s?|xL;hC)+gdvz{H3Z*CufO9Asr3*G3@}Uqz*)qPJ~8mO-_8ekHseoS5>QML{Cgk3~*` z6kmcAkjtV#i?0zYh@qfH=#BAnq#!7y>t#hQ3@tD+T~o?eJv3*oo3zZmDfHgAb$U`? z^A+S{OF_e|;L?FO(kfhOM{b&;ewvZ3;EN6@rWT*|luVuo6{XPl3_K3N!4nN)A<6mP zH=T3Zf!Y-Je|Pczu*!TQu$GA?E=&1!c`X5f6+Rdrs>gCapUh*Cv^vFQBb^nksiPt_ zk`<!<6w182}gDw8Fpjj2;Xb(=>lOSeu1Ss1k2rbrA z!sc396F!hx=;C@5RkU;9Hm~FAC!bzZRwY4Izod0ZIv%Bl#3PvHdb1?MG5HN#OrFa^ zWF4EC;?D&iCqdhFHtYKn+Lw^Yh@Bh#-+SNTY@G%bdm};#T zKt^#yLNcw-o<3&xb~-zItVBair^ifsxmeB^w2lq7&A9t~np`whB&4qDq0Ln~dMAUs6j~uJgtr3wn9Di@_FS$@xHjwQjM# z#CaxCyDkJ_UEY;U-}el5_x%3srm2zznMvP@-pWbrOj^G%p?P-CT+9_igP!s2s&CIR z8?y_q>L<0;wCR=T=U{Xs9*Fkrt;I~jfGbUzA>}WZ+UQb7v94TEL{ ztt9U%`bB3Oa|MUz8eBxq`N}B11n<_xaN4$|eKe7xtPV@&Jqp;amK|#XDudOf@!=7A z-;Vg!4%9j`Y*jg3nmp=mxRB?z(PZ^H}d+L#H-BI68$pS>()Wsg zBwN*(ye5}KrpS^YIgdd4?AG|c*VW6!QAD9w~CXCo+_g$hE4G4WXhAOgVJ R?Uet&{~zE#M<{wb`#*xsh{XT^ literal 0 HcmV?d00001 diff --git a/content/ko/case-studies/zulily/index.html b/content/ko/case-studies/zulily/index.html new file mode 100644 index 0000000000..a9e480ea97 --- /dev/null +++ b/content/ko/case-studies/zulily/index.html @@ -0,0 +1,4 @@ +--- +title: Zulily +content_url: https://www.youtube.com/embed/of45hYbkIZs +--- \ No newline at end of file diff --git a/content/ko/case-studies/zulily/zulily_featured.png b/content/ko/case-studies/zulily/zulily_featured.png new file mode 100644 index 0000000000000000000000000000000000000000..81179f36d293aa0693ba9c38e914c351d2cc3b6b GIT binary patch literal 7953 zcmV+sAMW6ZP)?{BF(?~m?uy8Hg_?Y`+WX&kD)2mK7->b508|L!q3j7uz&AkbB)w zf}9y)zW7`fV`k#qj1^nq|q|Fw%+G z{NOc*QokJZ4!rRtnX!1IS&^9AK6qDHtF_qN;S!L#$h>Yx;+qM1r1bpg>=ti^U}~%& z5qbb}@~zXB`NJTzEi|tI!jUhljCBa`YW#;1CtT7>hX#}4@SwXVD^Hwhxo0}2r0o%N z3Fo*JsgvFjT`8;7)@m4Kh4ZwjuQl`A5UUERb4K&1V4$3@?DJk3boN)M>wMD+aN=Hb zXYT2Q(@|%SffN|Dqzg};gUs9PLdF+{8yN)V~BGLWZ&wSW2 z^^%hB-GAtz$DSk5L1u6H(AAA|>q|;LNlb~Aob30A6QKfudhMc$vgNx5?>?4!=tOqM z+`#!w=CzE}5n+d7eeUbX^eT02epX>+fxA@EAeFRNLviJCcSJX=a!Uq@|e;X>~UU` zAV>bxv0`(@mzIS-CRyrm=68D&Kky(q@-*hPIxNFU4RD|SrD2#+7@6#L_KbX)lO5Xz zzH(CTSHM^s`SdLF?LZu0cnBi#gxdlrN-DF?88Qi6Fd)uw;sj&Ck#Rr;1tKymIr$_p z6;>$b9I~N*?04!M{mOazWn8M=^5q?|Z@7>idXn?H5Qw4bLN-4yMARSz=;g!!A^`xQ zcrEsw1oSb&mBEiyD(ivS4G075{Cwj>Dv-MK9(>{duLneVPKHs>cvdEElNoE$8z z&~E}#rof8CjECuj-IiZZB)%|oLxeYQGP>+nTp9!b4TBP;1WyAkKQ}?b#RLjn#SjAX zFx!%T1h~N=*koSoYj_WcHbPtajl5!*@(PDe=P_NYrKyY+WT2%*heJNS6-irUeE`U{ zS**|54FDmfiIYP_wv6Y?OLB=kVDSoNH2|i)E}K#y?Ahi!IT-o!0y8mZKSSjVsxZaT zf~Rwsd3ptbs~EI#usgTiS0Jh`FIN^28Qs}!fHJ@v%5aq=XTaN+FPsz|Qf4?**XU~i zAmJW!Fl)$)dDdU}W*&EsIyA`1(x|M&f={i3zabOglvpv85Tex8U-WR{uS*%0~c{?xq-tm_yV z002lWnI{*T?{CmArc+LufHFseu*$7Gt6LsS-It4ZL=GPk+ zm{<#di^A6@6Nlo?iR1LNmWeF1-aoFYVYR{H%4nZhg}QE8NE}Q2c0uSv0B|z5#e(wr zf$M0LOqkEn!OZid!ST$~?cQ?&X%^qGH2jw%-j_)NNBzr3BANZ_w|?qMRo5&)k+H?s zzw^MizWRxp>QN3*0vetMr)_2L#|jb3w0EK}wj%odAX^C$D~4Pa~^B9f5RoO<||f&KMtduY-a^WkG2V+DGEoD0H7#JCM(<8+Q+bh zqd!a+FZryXlo^t_rr=EgNJIdH01`Z#Vli>(MEaSyXm@$G+5UB>y=7C`oeG6FMDOg( zJ$oee(B9;~4CMAL3;mUfk)iAuskQn$$5()$#=KBK+7;Ob>&krf#!0W%POs-37&sjJ z#c}7wfHr$gv#5t z)WtCD2mt_Kz)h?%m740v#<+XvVDiBvbtHMu_SiQ!m;W7?)-3(9YHic8^dl#nCtp5! zbV=yON@Eie=L~eHgW+*M<~&qCKXGX~| z2nw}kQ&U5<%s4raGSncTbewcBq^@4Oa$Isi7~Wu+Hu$i@z)X1UG0`05OY)y`Fv`_c zo5~lpW`Ez3eIkL!_Qij)I{Yahreb7K@W0K2Yiv1?a9jJF9YL6@Dj}5xeXX71P+7Sr z5)Q=ztoHoMuW>Yy__Vf#p(}U(%`Nx;`{Vl$wgSQW6$@{=;c`g7W_KSWN33AkMriEut)S8U3hFP_9=hXXBGPHc0?jGXPWJI7qaE|fL*V8|2wI3;3 z4Q$XZSrhtPd*($)_DowL&njqYy&M6DmF+YRmO>pFo6PreR&_Y}NWcA3am&0$JFiMx zJm4LghT@%l(5bcKR2r4WucXd03QjL7MjFivh-G(W?%A98H;;VW??WJhrNIw(=XU#U zXU#O6)(B_Tt~8Vsr*#@D=5d*4UNx9)9)M$=-V5=;{ma51iK>gyPycIJUZZZ1nBg?u z(j}tq-T}`W3iiQKTHE(C0uT_BQdpK&SrHk@C!R=mbPWvcv^UPF()EH-5SdJ_wY`^d z21HGb^}04jJ0Vy16Yb!=SEyfBXYkm10yW>CVqit!3?@W9r#OBl+o6FQjoz-y`V5egnsLEL8 z+k2#hZ+`oy{fVp!W`0$M9h3wBNT;T2j~o@?tAGFT(URMD9Qet7j~TiyJog)4y?s`_ zkK=L(2bzz6>-!Jrh9(^6%U`&4>7u$ZZyfrsfX~&OifF z46U-vS`y~!PoUpZUSoin~(&^&+rN!a7coIqrhf_d){L>_(^w5O*ChGqe}xn}!HLAV>h=!;fw~(A=hIoCrNv+;rpRnp$wJaJoVTAmwOv zV*ue;A9TT&KKAgBy&2BAu}WR`hhxwaOm_~brW{NP_7 z>>o(!x*=R8XwvPsy^j?QX`EE67Q-YS7ZoWca7Na4exf{9klO7h4kqstK&WU83`Sv& zz@#IkC79jmzSiTtT(7@lS@2`NR&q&bK6HHBj#kUk7*NJ`KK9|ih=eVFQ93nW4^1$I zG12o0P}rwDB5>B$D*)s4?`((r`%#eY1h>KYgI8ThOpN{Dqizs5$YA#gV_%ptPes05A|s{p8-qEWQm?ytv6RhA2mK5y`flCk+#K~_^v8b{n9G#zV_AA;# zMuAC}1_F@f_b2wg-l1z6A!Ob374O;<4o>29&*X|WwmtIG;6$_F1Pv+~Fc>wP z`eQQ7$byTnVOBzXZfEwzF6UYBjZO6fq1R)-b|7(2#8`n;!igy%&I6-sF=5zyx(J9<7CA#m+8D%yYP;d8tOUE6z6G8Oy%Dt?%znx;s3`7I}j}CoYd{ zANi$Kh8>_1OT3T_SJ)jySSzjf$L;$x%q0Fwkr9(1Fqud~ZNX6dOgf9h z4k5o94{wkZ#3cM=!>VPxBq9#eK`1B4h5*PL8WTSJjnM~4e4+Wi3U$G*Y?8Kj^gZ^6t(Im2Fvm&0 z_lon%%fgdPTU%4^dXB0nLWsVBR1wMU-je_%C7LETjeO@tF|!#aw7*A`R-U#;#E@km z#2DwDeSHJ5R83XHCkFrkI608)?Tay$li#TFx19_uUl6!{9$uC61{0#i7BQ8Fg1pvb z^(qEW7p@d|LVov$4+u~+MB9_~b(Pm&d)ee;b<4{`hycVCRef>Wo=Y!Tr~1X9q_g&G zdzuxl2+ysmuePkbu`-#G7m+DkV8(FCf~H-2+jUjX<=}UZy?E;_{zsw!;JFuGu|3dL zK=NsWVqrmyOpDjoYHy!wzNcQhD9jfPQ!?btFOiw2p54~f6Vp^p0I1Bq?UrkJ;jOxS zlt+W1km^aUaNgG5)7jncEB545+f(Trs?zi5?VDF91)K#_iN2JDgQ+%dTERJ&k~Cf4 zy6yF+p52i@JxQW%ue|=JXLss`f&k>_V4mzcLtwVw*&TPS%gC8aaDNB{8r<4?ZA)c{D+6uExQ!b#SyBovvsYpNnQ zf9T?WyZ1@cvVhaiAAI7q{ReNj`qHwph^lhOal3m4|M<+dS9Ts`iq3}CE9jHoWLS}X z?rTTV52xjE;+`iFh$~Xl86V_&uiI=vWtrMgXROzF4Um6^zx(~O@ua7jDu85&*_`|E zV_Sg0Z}K+$X-`#^c*6Pd&;GNrBG^z<9Sm9VM5_77 zao3ZIYDhvmms`7hb|IxT;a0#T+N7ZNdDn%TckDR0_x0mez$9kw+TXEf-_Ija%L-_@ zOg5FU9S^u_jL}M^?0NwJ0MfC{!^bj@3&mrkT3&gzu|BNL3G&$*RtyCY6A%Ik$~q#W zF`tGwbfn|aC$?*rM(6@DAOr@?PziziOeF-Q3a04Bc$Hw8+86)FUw`@k{8-pR(Ig{t zU{DOijuQY8F~${D1p?2`ENH5}?W5Nh*r!1R27rWxg6a{74S6~I`5{Kg0i^F!#_+%2 zdi^~=dt~>%qpGUwToEJ_2|FIoaYRH#RhfhwN#fj)1&UCN6+~)I-rwT>Nn;EpHJFz- zn(xwZ)Ixzy+cO#2qO&N#cvd`5{PT}~NkRpeA%GZ4b_&wZBZ45L=j3ue2kB?kSN!ig zKJvYLe$&?<=ZJ`i9ER*6DbVvgDRb|*V9m`pTxt4U(1HMVE=|HFmk3B(R0e=V3;|Jiu1JT%*3BQd zV$1fu%?FPoWBF4U7P)1*+u6D|eK$cc%Bq(IJ{nXCF82t%=D{BkJK3F=P-QV1T zz!{tD=}-cYMCH-Yye9t}g_5b<%iH(vJ#b_&p0zU`ATZ8N-KeV$ty?>9&8nsTA;JIv z(0;tLyKhiIl}N5$xlkGFRv@B7M_W@#2Y{KTwq((qaRct;K&+*$6M+GUMGG54!D2qy z-oC`n*A5&!+Lq4P!Xp9^3K<*MEL*i=Q7CBqFOEE`%i z3nBmUyT9Cjs0%f!ua16wEGq>Kr~d5zwAWRyuU}|=xVZB%BilQ(+iOk#+t#KJh^Q!B z*EK`eG)v`Bw|GjN$cgN-@Lv_D#HNlNiSK41F+Xq}`ASZDqw2pPPIECi z%b2>dTNVbdRlq9LsD$#Q{p$YAgAVmq;oOL_a3pI9u0tms+3!l*;AJ|CDsW~S`{c^B z3vo`Gri2jmX+ECth5`|7+Zhf7&6{NWSV19WVs7w*1!g4a(7@s3L!I_Esknv=tStM? z(B%gbv%U7N_S|!6_s}f;!luCcRg4Ho2uN}Wm0E5O5~^aTRNa#P&#w4OQEhRfb-Bf7 z!C8t!JGYQhCgKT(01|a|yxR6*MUqo*Y-Yemks<|n1YS<|c4c1baGrNqmK3JT;JUKU zSga0ECd9FW$zS(-yMnBIW7(ZSHf%- z?VAFBd~&NJn682J+zo4%j(eKl;|Kwx%HlfX%Ahvak%{iyOTF3VxZCbg0wo4yWhKm8 z5xRX%^z%cM97+A`xbq@_R;z4U9s1O0N&vvc#DIH1f;0L{n|fLkkB0T748jI4YfV1_ zIOd#~LxU-|cXaU>Xp5|mEHN$z>2!EI4kUj@CEf!%Lm(x+xOLB#7x$=|=6Nz0FxRaa ze}dj;*9!oE6ct0Y>Up*5d4#S6$CEh%Kwt_}IT#~e$S(W&cIP=@N<_|G9scC_*ggPT zWO7~_QI41c8QHfx_Pw>`pB?(9LfhDr+jT7a=sff4(=wvRWdnrnn57c2`GM;)qNm&4 z+2w9klYxc756-wV$_3y!uItLq?%t=KeP!Fu!%VdZnDE?Befoo@IeySj>@!AEIj9^@ zjeya0o5aU%?)1ua^Dv% zR$i-JF!W7IwAz#s8QnH-)Wjxj$5!o1=XGq*~>if(Y*ZmtQp>>B+hd z5GqU)Oy-iw>#jb3`I0&L@>A+)NzszU90;*AaI=opJ`(5++Pi_1KnDN-sV(#S9CbEW z7Y{uilCpT6u2(fD?;p9^(-q`_n{W^Nl$Zi5qaRgeRI0KkyJeEw{^u41)X^JH=SWoy zMb$l!((|spYV$Skz1UZNO04we_IjAsq*1N;2Xv+6vAAqeq@{ob02z7wSmvQ1HdUyj zl|Lb=_1YCSrH^JG9{NgxOUU|jyL__^vc@`Xog~)h?GcchN)_H95C{*2WUlA9xpc@- zzVMk3-gw<*g+i{*{6=;}NO{K?Vzs%BqweR=q4vyH7wsUHF)#tBE&YT*5!IU5n9Fk! z0781BlG8i0&(AZj<`4+*2tzjD>=BURkbiapYm5zT$>$wP*)nNjY0*z_JP={&qT0Gz zFB%Q3THd&HaZ@B*%-MYAtPqH-9OQz{HQw#JddP_5ow=>Zl`3U51I>eYuluqDR;jGY zr!b6l3R4`2ci7L*(cc|amU2>4tiL7u_yY3-`I(2*1twP8&@CwqSeY*{qkxFFe&hoH zG|BmPDtEL3k!E1Xt>ctB-AfCI46tHvE3w~L^2nfHwtiFO0}jR_C57httEin3PqGHKYcr-N@Rm96!pUabOeG^>&;4ibx>4G&RR21 zpR{-Tyj`t9(k=gy`sT)zSP5&58jjhhM})ZCzdw18gCazgo60_`v*8258mmP_0wKs} zKU{+CQUaxFDm6pCVMhA8$ftrL>~ec|@<%cG`nV(tMs!xCu#)|rGapl8rBYj=221zf|7Vr|o&)*c75-S=DAr}BZ=*8sGow2)4*w0m}>(_^GUmyOw z24hSk6U(c#HHZ)shsSxn03a*7r2v)5dVTX~6{0|RW7!wx>+fY`zBc&Xmh^)XhZiUU z1}l^$XYDNFOv02@B?rrZhNQc%Gxv%IFxPx<3_^z($2W$FY+Dc_`+ngK&B ztkw~Uh1wDZ+@Fk5(6GG0de7m^Ln-H2nRfcuy#(ZXoNX$z=9=#*l$L{4r(aO5pO=+= zS@%SL=2cUTntbkbUek5XFjcdX4gH$kq?=}*^`6emOD*jPT-@PW}c`QBF{ywSDd6xdvLN8CP;qK@IOP#j48YQA}6gK=poIpMkf!SQLt zIjESy>NZt;{$T3

3?y6N?ySl13L{lAvjX{P12M33(q$sETJ}>sxSI z=;vlK60|S5ex4WL7C|GeiKLXe!+2CGXL`vv66ZYDzzO0KB^Qk&uuK2!T=2OO+T!Gt zzGy25{hYR1BY4Z?Z3*R?NV1yHCLC+Do=8*FB4*XvVJ)JAPiqI>iFV#4lZL_p5kp7h z`nQ6i@!v_1ic*p_ZGa~?`5i{s5oi^>ivz4hVGy%1CEMZglL-3(t6Pm>MJT1 zq74z@Ycv1fSoR{-@b3vXqsfl4qM%=WIK545fD(qHABfqADD%+LOA&F zh=E_);o249(Wb)ww2@-sGz~&AV#l(V1||BBJNl~nnyk^Kk+PVF9W0^#rE-m&z z$J;u>DiTyMOEw>4P-;Sj@Y_L4$+i^FjBAtw5p)Kt9Qa%~>1ngTEN_$|g`A3%-wcx# zbToq!{833cDQ%TJ*_qI>9ISBdzT3+JaN^BO$kdiZrN>XCS>;0xk&}pn<ALL|sch0P40}^}td7F*& z8`!`fw4iZSAX2uo3UT)Vd8O(-f3b)3!o5cMIG5DBbueZvry3e~yTz|)WFyG+$J=cF zSDLLyr{#q%cj0I}GKr>!gQ35N2vl6ris>KcYm&ID>e~yKM|u-7@cb>$D+Gg?E7jFm zD_FluKdw`jpZpuN-cRsHPV?S(!~ygTLHcz#z;&QBA^nI8q12G@fv9tWuVTjp#TE{p z*gx5i4*y^w1G$Ub`u8rJp*a)W{rVHl{9R6$NKgQ7-gQ>%&|GVkQ zotGjfW97f1Q>Z~N$0d=WgQu0oAT|1aZ9W&zeq6{%KvCSv#P|^st6rz{B5L%PEIM4g z`{U!t-c9Zp`_%@kDxRzowWFsAD-r-0SUB=&H$6ns@E!%?NR8hmHW|AQ+gbg1(YZFE*<2)51&s~>B#=KjMWLu{Mk1cBhInxd?*=zx;nD;*n!B~=!JPrRP%^^o69VKj>ir+nZB z4lLqu0Xu|AXOvQzPl{?^H|jajpTbZAm2ZfPQA>t4BENACM@S%Sf8yN#jukeI$G@gX zw$7rQ(2lApU9@qqPNM5G<0OW%zK$lw0OYdWo7ZLd=H<87xj3hiF(uv1qR}k_vrY|B zM$nPd!cW?#u6>XL2zi2B2k;$G;<&%aOZ6M5$9LocU8x!QbzA=A+@S<-&29gQV^A&; zA70e@Jg4rn=k1uNNM(qkTyK8Bq6O8y?(EPsqDRV1J?2@7?i9mttmOtdW6zWl>#rIm z(a&Rb1+XsRqDwWkWHq-$rj3?>@r)WJ{wW2TO#27eV(8dEV$9%YntFm-;kS;!CWU3} z{wh;in0y0g5B~rO&n3r4M}1G@D%B#}>!6fYy*Yhe(eLTy24mv~UEGfFq?G}fcqQM( zc=;}liXsBpW)tgd0~)y?W9NG#yG&!Z=NU)%ZQ#rtzYauR$J3YIl! z;=f8+#=d(*&e)p1_EsTMzyPl=V$NLBPNmC_@yW(TQ7~OVZ{{n%Wi!5D4cgPTuS?;%3!e3K?W-w-x&Yz>+Z-MDcD-UO^C9pr+d(&hfwKXd`VO- z7pEa7Ux}BhnEU@iuHhIr2L(74Mw*_7vR@&Rj}cD?W=q~1s|Fq2NVwY~%YpVE!H4ei zmWM>qFNlO*D8*t$ETQ_PKWP}1(JF*<{nEVs_?>j_@KaAeD+gyS-_zMC*Z&Z`UN*^p zG)-#^FglI?Vc%L;z~c^&rmetjE2g|8u0DVEW$QyD(*y0G9}t3&&Qwh-yRu-+X>tqe zod({;@EvA;R(xSovLecM3`Ey@{cd}J5@tE$gv#29qBWvWiq|@I;2YNS^kJ^09C<;k zZL|b0{R+#@^t$ZVATn}WvTXo8<+Q-KMMvkAtCZFU-jiPSkZr%=8;cFB;Bjbb_R8vw z-~6y+o%&0_be7qc=53dl=nZuT+oe)xLH-ZxPW(*sd5cY%7IkXg&RGTWqUkw6k3;ki zxY=-l0NzR#QQi!$)K66_lDBtzi=<0K?>XSs1I>AcBOpu_OV58E5BTCmiv;m4_<8qB zK$?uD7MJH|LoF#DWq907E~ZYO!ZEyGq$AyKkniyn_#&@n2m~O^Q&@Hj=^T^DckAKQWk~(2M^lIMPLzJq8Jmj zD*Olbt}p`Xu5^<+DKWFP(|QYA38T>(zCRI)_VJl>UBVeJ0NGck>&{@=lieD~#?W^E z2kG{fy6$f5T3&?dj;DPK$7v>x97~M23D*t*Hf(0DP;WZ|@G*uN$rJV7VuNsfwor{m zAjT>2=>5p?=FbphzLHD@eVMGm3K3d4zn&jgR-1F%%2;^*b#8|X0> z{^!I46^oE1h0D{{%FEYjViI($B_$o>5zCS9&R-z>%ExKU5d5*;%#XG9-a&PqAZy6?TXhsoForL+HtW$&yuV`%$j z_QwwXs%6koc1o@g^K;ArrkS7vAgMOrLbGF5ThE^Q~)8LD63iQD-x#M{K?11&Ok> zu~UKMVL@7m8H`51WbI$_G#5d|)^z>i)c(YWx`DRjHh#&N zv~oiaA4kN|Q=+Bf8hxEkx3&Dhw-`}U<5$aa7hrBXLQz z%}JW#1HR1t+?hve_hpOWEJJf{9PCA=t^Cty{wV89mAn=p@kkX{2byfiW#bByVvYKQ zPWd>#{2FCiPc`g`^bkI=?&Go>N2KDYUo+0MXWeBh&6-H8tEw-a?;6;v6^c;1KTV(e zN1-h1l})5qlRI~otky2Sk4u$I(}YaJZuGkjmVeCHBZvhlXr(l&*$_D0F@1(LLnWxOa;F-p6u5Xhgo{VzJGrJPgnVgu%;o}@|WGW^yya#8tA-DMn z-v{im!lmdF;bf5uSWl()Y1Ku0AjMV8k42b(6%(Ye9q!(HI6Ke?BnMAbRpR-JvAvVZ zZ;mq7eWfu>Utg6R9!I@;oyid|F19jb#=B~7p0P;Cx9VdWamj9~hAI8mQGM_ofX8ju6+VV`>s!D)I09za6kSe*9X4e}DE~*h3 zBZt;s@61wzgC>Y$$5n_|e3?cDlATBRSbU5ECGLBT8zzM%U~vPk^ON}?W&%xgt$NeH zgV7}(r^fbZl^m3d{<6SojYhE_pAwX7?MUJ6=$JjGWtuLWqlU}4`wDF?Yx#`x*~t4A zbJon&dNVTESJEDXhF`2GCL{*O_hMpWm*QPwwDUmJn`7H$RS*%NSliF3wI%Ss;ZN7? zF{}$k54ZIYJB38_`&LRS7pUjQGbnF+F27ULXw zYJchNZNYC`G!AW;BmOs+#traib|WbBMZd6rcX15WT9S?7fz<3q9xR8-Uo8s+W!0Oisz>!mn zpB}modcZ}A!X<8--g%0%Yd!yBaMQq%-krUR=;K14*&SM#&u05Oka}~!0HZ~;yhy|6 zw+_tD!}lbzT%K z)cwv)=*bj+NqMjabCmyR?qc;g#&-|224TeC!{n+CSZzOrtp@y@`D*134^gJh?@5O^ zEHCqfv2X5EU^A}=q+@Qk)Gsb>j6a%CROF1v+&5>bN9#>>&4!!*%8x=&(%%EzCI@8- zHf~yZU76+8dMP^7MskFFVq(sNW91Mtwf-ZiC1C(Rwv@$f?goBB#KBIy$b6zk9zbDW zDgN)FqN}0yA*7kW11C%h8UFjKguM>SdAadnMiUj%mwhVSNGXl8Yyso*C87m2J+GnT z>)>`c&~pc>lohrK=6@!dfR)ZljZ520Qt#w9qWRDYSQmuEnIt-b5x+!*a1jk!QoKD6 zulx(_ze8=S_Dr8cEIcz$=H1&IWA$R^;CyQC>l^uB`;(qt-9cxGKZ4}XR!#&2cc@s= zF@EAA&Yw(~lrg}^a=C!Njr+;w{yX_aRu0~D>E-hsPBWQ5H{0+7H= z6Iq}>t$f*3>o{LzdkKZ?%I+Wa)f1T1894>}3tX*@4OPXs=!|99bSkb&N9*B-15cOo z)zq468*}>vSIK*hZ`K%_oVXr*=RwO$m?D@eXwr0~AVmmHpn^h^@9T;9LIWoXio(P> zJ#Ar}?!P^?L5}I);3^!0mh2RM8J&%jN{mVe^L(;p{?We}2{w)P>%i)$>K51uG}XNs z;!w&xVF+@iCQ~r6L2|GnTCNCAE)^O0>XQIG|M;-QQJGo}HBn3N$ZLm&DOujmei6Rt z4(;oDLnU;lg2<(9H{eOUiBea!w0JWp{03kioP4g-<*)#8s27N-)M=JyuKr6%vLhuS zL$TPB!^+D``;9E#5G>{tiIO>rA@1;HYSoOB?N`?VM4qmhuYzO(n*dJZ)tWfJ&Pez; zV)#0+DC7_>3?(1^x(EFJ9?S&B;%2kQ3QrvTsFPNXZ@jdp;~L;}s`UfknH6n%rFb^x z2GWk0o_9g8^=mt~9At&ffQcWa$nc^;BSM~ggzKiKf6=b5(GaGRgS!cqJ)~D^PEIRX z3%_c6KnHa^koO!vPE$tltl8`r5PgE#wmO3+S~YN%nEW>9?`%J4 zX2ovranuW`ekRF{U#pWQnP8%XK^Q0KL-Dy}&&=FkNpY#*sSr>>gAnpsxv7dwL;_eA z=*z)Xi>ri;Se9H#!)J(%qW9^l#@Ih#70gKuf_xQNY_Pb~$*HeqONYoOSON7veZP^O zNX&rkUopkB7lEzL>ry;m7m1uPqWf(EQQwmhZh}O{F!m3n{sQK3WsEV3An&6G z)21PS`aOjs_I=xWXKt0x+0MlrZRs4^f4y(CN&oIH&x~D5DuknKatY(PTyL>2_H?)} zPCNpDl&GwVTd6b}t+~EM|F0BEzn_u=nk>{&K<2j5A-g(I^J6r^5BEeCX4$UTP5?+C)a+ zO`T7Z)15>LP=Qyz_cB@I>c*qXpurOz;u{$_XS4Z*pn#O**GFe%#$wM&>w(9OyvpTg zfCu;Izy77Yj^{ZEpobQX&8hvj<4yc~My$2%;^LL5>HQB2OGOMjB-S2&(Wo8?wsDFR zb=HWgx|;@P{lkFl*&Cve@z)bU)-f?X+_LpQG68{zz^3?|1|*O`S&LNJ_*{%ga(!G5 zf6|$7U~}@h*gc>6q?`kBMwmj0btpbf^UiZc>lK0F4yzLD*E8~Sv(;(4kiS%~k1CrZ zjYRW&!oltZyUZY|Pz+t+Z?iux#fz;h&6-n9g8RsPqVg@ZG}3wK zB2b*O|Gi;h5AMkg315zT3uWa~D~Wdxm+L#K zFvNy#6amhj_D~HQEFGo-vbAS3xLl2PGvydNJeXuKjPgEzUSq%)9H$?9ScU_Rl<-py zU1L*pdAFs?xSjrXGPo>0-+gtJ82>_VH%5rbQafu@8YQ*0utBkpC>g*G4Gcr2h9+Tz zqof1Smc7VNi{MB;Y?$w_P}4;YDDU^w#P)vBVX))WnI9;al<5Dy$NW{Gv%PL)Over) zM@JZ6arE?UMy7-?n)c9a73c)zwFCJ&Z_+C8`dyI{&nhm^0U8{X<=bs1jPmB2HshD_ z(u1nNrcoFOon6_wrlKC=$-TYs87QobDqThsPI@|;%DiGc;|B&f&6Pbp5M3VNewu8< zf&0n5gL=Wq$aPRSoY-bXhwxZ*OZCBtJ?yVO&PDdyU6;MZ4@(EkkmsnGNuzz4Qd$s~ z+RxNy*^F>8!Sawy^?g{!a#N(`-Dq|P-7A-)F#qSZQlGz#6PsJt@4HksSsgR}rj0az zXzURx%lW31gLUF9)R&_6G8ZMzN1(3L!$jt;5Ni-!9;z62*V5GBc9)E)^_yaehEODeWq zC;9o7AgB|D^iM$hcH{1!Qv6|OsFOcw-`M&%~g&(3^G&b<->R@#BaJp@f>rA8oHEL zd+bf6w4}_ek%Bs0h^<5EtqP`H?N&k}^z3*~Zxyhb94bDTsYkrk}g zvB_kGJXgpko-WiFE#6|k8xR*ggy|N?HnB_~&<=kAN+o#GA5S6V_brcou)DO{@Eix+dv?Qgt3EJAS zgrw412#ofYtWFEX<-a$u>~a3~4$aM%*>kR>Jo9IXw5f_VOJf#G`w$=6bMMEGEF1Ei z;iY+h7}&sUUR~Ba9UR4Tm1J!)F*|TCxm<$HG%;Tbc4s`WO~F1ubJ~-cto|k3+4D^o zQCg0~Kcyx`8GCkkU$3@xDrIM)@ZR!gDbAk#k!fl|o>0_8Q%}DJq3KfTtT6n==-k7u ziC>@a;(UYX9Xrmt@!fGI5mL+B=TBXROz`a6*%n0a9Dx=7AZFqnQ7C5!<-Je=yU2C; z@OZP_IG%8gd>4$F15x4wrbQD>%u```i>@9~LG%N!(%y-0M222z3h@RxKkpEy3n;hQ z-!5p(l)ex(xABA5`Hwe`DoypsT=v*P+7!uq$`j3SCF7U=O08dA3ZP`UY37X)#z>#b zCuL9}!dIVk9k{n@J1??)!x9cj_=ZZ2l;P{O=X#jn{3~^;gx2;Z)Hf<3H^kccYD0_e>1szp^vgy!tQJtUpQDm3h3wcz{q@qQ2KV6cpT^ z9w===0w~ho;w^q8x%53nnu}75fK+1`(fj~5F#qEb*Ja(3J7dn!yAwK2yAv3Mt0a6r{O`{Uq!)-a@w4@_7$r@{L{PCWHFqxZ->i2PI(8lUbOxMb25|%nJ~s$ zhqI@_nJeBl(*?Kw@S%_HQ_|CtvJhHrnD!2=F|SN+SaRC!2Qeb5?BJXXSm;SSFpi_a zRqo($tr6XM7O$hQXZOGIzV?WUX;EPLAdh82)-5grMowwovFWU3Jw0o7I_Bdjsw~5v zHrcNZi#)Qm6i;p?@POBG`&r&UlR2HgB|Rt>bMe(%A41vY@*iih!c^(r;i;(dD=!EC z5*=jg5Ko<`n9k9YKyYn_8%`>{13`zZfaN9DdtW3C#-pw9A?OTVJ+rlm&K>o)oYgo( z;k4d+Q!uf5c-H+eJUhz)?ezk)FMOX3OEx7I1TL5eKpS35ly?ST!1sDLct+|4AhsyP zsJg`kyNc4Ka3rt8+YfiY$1Sr>ZJmSiaDuHW{j`BwUZ5L_QiiPu@~WGV$+HA}#nI_J zQ|%H815O>4OymCIhmQkkI|tD{*`h_o~c2pzp8~)m@E|4?H>&* zR5rec3XSB0I*lBX%CMtLnKK6YH~M=*Rupaue%|LpfJhJU@OozA`dNX%;3_ZkOhg z=>yP5PIRKyIHiBb?Tj{Bxn3*C!ydwE1RkFw3XX>DU2smNMP=gBH@2T-%D3d2ZG3Qh@feM~$_ZRRzDuR;si_0fEGHccPRPuuc ztVzR4pxp?v45O%3?5H_2gP+PWbO9>OxPsHd;;PyvnfHX%bku{(^402Z1yo|{F}} + +### etcd + +{{< glossary_definition term_id="etcd" length="all" >}} + +### kube-scheduler + +{{< glossary_definition term_id="kube-scheduler" length="all" >}} + +### kube-controller-manager + +{{< glossary_definition term_id="kube-controller-manager" length="all" >}} + +이들 컨트롤러는 다음을 포함한다. + + * 노드 컨트롤러: 노드가 다운되었을 때 통지와 대응에 관한 책임을 가진다. + * 레플리케이션 컨트롤러: 시스템의 모든 레플리케이션 컨트롤러 오브젝트에 대해 알맞는 수의 파드들을 + 유지시켜 주는 책임을 가진다. + * 엔드포인트 컨트롤러: 엔드포인트 오브젝트를 채운다(즉, 서비스와 파드를 연결시킨다.) + * 서비스 어카운트 & 토큰 컨트롤러: 새로운 네임스페이스에 대한 기본 계정과 API 접근 토큰을 생성한다. + +### cloud-controller-manager + +[cloud-controller-manager](/docs/tasks/administer-cluster/running-cloud-controller/)는 바탕을 이루는 클라우드 제공사업자와 상호작용하는 컨트롤러를 작동시킨다. cloud-controller-manager 바이너리는 쿠버네티스 릴리스 1.6에서 도입된 알파 기능이다. + +cloud-controller-manager는 클라우드-제공사업자-특유 컨트롤러 루프만을 동작시킨다. 이 컨트롤러 루프는 kube-controller-manager에서 비활성 시켜야만 한다. kube-controller-manager를 구동시킬 때 `--cloud-provider` 플래그를 `external`로 설정함으로써 이 컨트롤러 루프를 비활성 시킬 수 있다. + +cloud-controller-manager는 클라우드 밴더 코드와 쿠버네티스 코어가 서로 독립적으로 발전시켜 나갈 수 있도록 해준다. 이전 릴리스에서는, 코어 쿠버네티스 코드가 기능상으로 클라우드-제공사업자-특유 코드에 대해 의존적이었다. 향후 릴리스에서, 클라우드 밴더에 따른 코드는 클라우드 밴더 자체에 의해 유지되도록 하여야만 하며, 쿠버네티스가 동작하는 동안 cloud-controller-manager에 연계되도록 하여야만 한다. + +다음 컨트롤러들은 클라우드 제공사업자의 의존성을 갖는다. + + * 노드 컨트롤러: 노드가 응답을 멈추고 나서 클라우드 상에서 삭제되어 졌는지 확정하기 위해 클라우드 제공사업자에게 확인하는 것 + * 라우트 컨트롤러: 바탕을 이루는 클라우드 인프라에 경로를 구성하는 것 + * 서비스 컨트롤러: 클라우드 제공사업자 로드밸런서를 생성, 업데이트 그리고 삭제하는 것 + * 볼륨 컨트롤러: 볼륨의 생성, 부착 그리고 마운트 하는 것과 볼륨을 조정하기 위해 클라우드 제공사업자와 상호작용 하는 것 + +## 노드 컴포넌트 + +노드 컴포넌트는 동작중인 파드를 유지시키고 쿠버네티스 런타임 환경을 제공하며, 모든 노드 상에서 동작한다. + +### kubelet + +{{< glossary_definition term_id="kubelet" length="all" >}} + +### kube-proxy + +[kube-proxy](/docs/admin/kube-proxy/)는 호스트 상에서 네트워크 규칙을 유지하고 연결에 대한 포워딩을 수행함으로서 쿠버네티스 서비스 추상화가 가능하도록 해준다. + +### 컨테이너 런타임 + +컨테이너 런타임은 컨테이너의 동작을 책임지는 소프트웨어다. 쿠버네티스는 몇몇의 런타임을 지원하는데 [Docker](http://www.docker.com), [rkt](https://coreos.com/rkt/), [runc](https://github.com/opencontainers/runc) 그리고 OCI [runtime-spec](https://github.com/opencontainers/runtime-spec)을 충족하는 모든 런타임 등이 있다. + +## 애드온 + +애드온은 클러스터 기능을 이행하는 파드와 서비스다. 이 파드는 디플로이먼트, 레플리케이션 컨트롤러, 기타 등등에 의해 관리될 수도 있다. 네임스페이스를 갖는 애드온 오브젝트는 `kube-system` 네임스페이스 내에서 생성되어 진다. + +선택된 일부 애드온이 아래에 설명되었으며, 사용가능한 전체 확장 애드온 리스트는 +[애드온](/docs/concepts/cluster-administration/addons/)을 참조한다. + +### DNS + +여타 애드온들이 절대적으로 요구되지 않는 반면에, 많은 예시들에서 그것을 필요로 하기때문에 모든 쿠버네티스 클러스터는 [cluster DNS](/docs/concepts/services-networking/dns-pod-service/)를 갖추어야만 한다. + +클러스터 DNS는 구성환경 내 다른 DNS 서버와 더불어, 쿠버네티스 서비스를 위해 DNS 레코드를 제공해주는 DNS 서버다. + +쿠버네티스에 의해 구동되는 컨테이너는 DNS 검색에서 이 DNS 서버를 자동으로 포함시킨다. + +### 웹 UI (대시보드) + +[대시보드](/docs/tasks/access-application-cluster/web-ui-dashboard/)는 쿠버네티스 클러스터를 위한 범용의 웹 기반 UI다. 사용자로 하여금 클러스터 자체 뿐만 아니라, 클러스터에서 동작하는 애플리케이션에 대한 관리와 고장처리를 할 수 있도록 허용해준다. + +### 컨테이너 리소스 모니터링 + +[컨테이너 리소스 모니터링](/docs/tasks/debug-application-cluster/resource-usage-monitoring/)은 중앙 데이터베이스 내에 컨테이너들에 대한 포괄적인 시계열 메트릭스를 기록하고 그 데이터를 열람하기 위한 UI를 제공해 준다. + +### 클러스터-레벨 로깅 + +[클러스터-레벨 로깅](/docs/concepts/cluster-administration/logging/) 메커니즘은 검색/열람 인터페이스와 함께 중앙 로그 저장소에 컨테이너 로그를 저장하는 책임을 가진다. + +{{% /capture %}} diff --git a/content/ko/docs/concepts/overview/kubernetes-api.md b/content/ko/docs/concepts/overview/kubernetes-api.md new file mode 100644 index 0000000000..bc84f00b6c --- /dev/null +++ b/content/ko/docs/concepts/overview/kubernetes-api.md @@ -0,0 +1,128 @@ +--- +title: 쿠버네티스 API +content_template: templates/concept +weight: 30 +--- + +{{% capture overview %}} + +전체 API 관례는 [API conventions doc](https://git.k8s.io/community/contributors/devel/api-conventions.md)에 기술되어 있다. + +API 엔드포인트, 리소스 타입과 샘플은 [API Reference](/docs/reference)에 기술되어 있다. + +API에 원격 접속하는 방법은 [Controlling API Access doc](/docs/reference/access-authn-authz/controlling-access/)에서 논의되었다. + +쿠버네티스 API는 시스템을 위한 선언적 설정 스키마를 위한 기초가 되기도 한다. +[kubectl](/docs/reference/kubectl/overview/) 명령줄 도구를 사용해서 API 오브젝트를 생성, 업데이트, 삭제 및 조회할 수 있다. + +쿠버네티스는 또한 API 리소스에 대해 직렬화된 상태를 (현재는 [etcd](https://coreos.com/docs/distributed-configuration/getting-started-with-etcd/)에) 저장한다. + +쿠버네티스 자체는 여러 컴포넌트로 나뉘어져서 각각의 API를 통해 상호작용한다. + +{{% /capture %}} + +{{< toc >}} + +{{% capture body %}} + +## API 변경 + +경험에 따르면, 성공적인 시스템은 새로운 유스케이스의 등장과 기존의 유스케이스의 변경에 맞춰 성장하고 변경될 필요가 있다. 그래서, 쿠버네티스 API가 지속적으로 변경되고 성장하기를 바란다. 그러나, 일정 기간 동안은 현존하는 클라이언트와의 호환성을 깨지 않으려고 한다. 일반적으로, 새로운 API 리소스와 새로운 리소스 필드가 주기적으로 추가될 것이다. 리소스나 필드를 없애는 일은 다음의 [API deprecation policy](/docs/reference/using-api/deprecation-policy/)를 따른다. + +호환되는 변경에 어떤 내용이 포함되는지, 어떻게 API를 변경하는지에 대한 자세한 내용은 [API change document](https://git.k8s.io/community/contributors/devel/api_changes.md)에 있다. + +## OpenAPI 및 Swagger 정의 + +완전한 API 상세 내용은 [Swagger v1.2](http://swagger.io/) 및 [OpenAPI](https://www.openapis.org/)를 활용해서 문서화했다. ("마스터"로 알려진) 쿠버네티스 apiserver는 `/swaggerapi`에서 Swagger v1.2 쿠버네티스 API 규격을 조회할 수 있는 API를 노출한다. + +쿠버네티스 1.10부터, OpenAPI 규격은 `/openapi/v2` 엔드포인트에서만 제공된다. 형식이 구분된 엔드포인트(`/swagger.json`, `/swagger-2.0.0.json`, `/swagger-2.0.0.pb-v1`, `/swagger-2.0.0.pb-v1.gz`)는 더 이상 사용하지 않고(deprecated) 쿠버네티스 1.14에서 제거될 예정이다. + +요청 형식은 HTTP 헤더에 명시해서 설정할 수 있다. + +헤더 | 가능한 값 +------ | --------- +Accept | `application/json`, `application/com.github.proto-openapi.spec.v2@v1.0+protobuf` (기본 content-type은 `*/*`에 대해 `application/json`이거나 이 헤더를 전달하지 않음) +Accept-Encoding | `gzip` (이 헤더를 전달하지 않아도 됨) + +**OpenAPI 규격을 조회하는 예제**: + +1.10 이전 | 쿠버네티스 1.10 이상 +----------- | ----------------------------- +GET /swagger.json | GET /openapi/v2 **Accept**: application/json +GET /swagger-2.0.0.pb-v1 | GET /openapi/v2 **Accept**: application/com.github.proto-openapi.spec.v2@v1.0+protobuf +GET /swagger-2.0.0.pb-v1.gz | GET /openapi/v2 **Accept**: application/com.github.proto-openapi.spec.v2@v1.0+protobuf **Accept-Encoding**: gzip + + +쿠버네티스는 주로 클러스터 내부 통신용 API를 위해 대안적인 Protobuf에 기반한 직렬화 형식을 구현한다. 해당 API는 [design proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/protobuf.md) 문서와 IDL 파일에 문서화되어 있고 각각의 스키마를 담고 있는 IDL 파일은 API 오브젝트를 정의하는 Go 패키지에 들어있다. + +## API 버전 규칙 + +필드를 없애거나 리소스 표현을 재구성하기 쉽도록, 쿠버네티스는 `/api/v1`이나 +`/apis/extensions/v1beta1`과 같이 각각 다른 API 경로에서 복수의 API 버전을 지원한다. + +리소스나 필드 수준보다는 API 수준에서 버전을 선택했는데, API가 명료하고, 시스템 리소스와 행위 관점에서 일관성있으며, 더 이상 사용하지 않는 API나 실험적인 API에 접근을 제어할 수 있도록 하기 위함이다. 스키마 변경에 대해서 JSON과 Protobuf 직렬화 스키마 모두 동일한 가이드라인을 따른다. 다음에 이어지는 설명 모두는 이 두 가지 형식에 모두 해당한다. + +API 버전 규칙과 소프트웨어 버전 규칙은 간접적으로 연관되어 있음을 알아두자. [API and release versioning proposal](https://git.k8s.io/community/contributors/design-proposals/release/versioning.md)에는 API 버전 규칙과 소프트웨어 버전 규칙 간의 관계가 기술되어 있다. + + +API 버전이 다른 경우는 안정성이나 기술 지원의 수준이 다르다는 것을 암시한다. 각각의 수준에 대한 조건은 [API Changes documentation](https://git.k8s.io/community/contributors/devel/api_changes.md#alpha-beta-and-stable-versions)에서 상세히 다룬다. 요약하자면 다음과 같다. + +- 알파(Alpha) 수준: + - 버전 이름에 `alpha`가 포함된다. (예: `v1alpha1`) + - 버그가 있을 수도 있다. 이 기능을 활성화하면 버그가 노출될 수 있다. 기본적으로 비활성화되어 있다. + - 기능에 대한 기술 지원이 언제든 공지 없이 중단될 수 있다. + - 다음 소프트웨어를 릴리스할 때 공지 없이 API의 호환성이 깨지는 방식으로 변경될 수 있다. + - 버그의 위험이 높고 장기간 지원되지 않으므로 단기간 테스트 용도의 클러스터에서만 사용하기를 권장한다. +- 베타(Beta) 수준: + - 버전 이름에 `beta`가 포함된다. (예: `v2beta3`). + - 코드가 잘 테스트되었다. 이 기능을 활성화 시켜도 안전하다. 기본적으로 활성화되어 있다. + - 구체적인 내용이 바뀔 수는 있지만, 전반적인 기능에 대한 기술 지원이 중단되지 않는다. + - 오브젝트에 대한 스키마나 문법이 다음 베타 또는 안정화 릴리스에서 호환되지 않는 방식으로 바뀔 수도 있다. 이런 경우, + 다음 버전으로 이관할 수 있는 가이드를 제공할 것이다. + 이 때 API 오브젝트의 삭제, 편집 또는 재생성이 필요할 수도 있다. 편집 절차는 좀 생각해볼 필요가 있다. 이 기능에 의존하고 있는 애플리케이션은 다운타임이 필요할 수도 있다. + - 다음 릴리스에서 호환되지 않을 수도 있으므로 사업적으로 중요하지 않은 용도로만 사용하기를 권장한다. 복수의 클러스터를 가지고 있어서 독립적으로 업그레이드할 수 있다면 이런 제약에서 안심이 될 수도 있겠다. + - **베타 기능을 사용하고 피드백을 주기를 바란다! 일단 베타가 끝나면, 실질적으로 더 많은 변경이 어렵다.** +- 안정화(stable) 수준: + - 버전 이름이 `vX`이고 `X` 는 정수다. + - 안정화 버전의 기능은 이후 여러 버전에 걸쳐서 소프트웨어 릴리스에 포함된다. + +## API 그룹 + +쿠버네티스 API를 보다 쉽게 확장하기 위해서, +[*API 그룹*](https://git.k8s.io/community/contributors/design-proposals/api-machinery/api-group.md)을 구현했다. +API 그룹은 REST 경로와 직렬화된 객체의 `apiVersion` 필드에 명시된다. + +현재 다양한 API 그룹이 사용되고 있다. + +1. *핵심* 그룹 또는 *레거시 그룹* 이라고 하는 그룹은 REST 경로 `/api/v1`에서 `apiVersion: v1`을 사용한다. + +1. 이름이 있는 그룹은 REST 경로 `/apis/$GROUP_NAME/$VERSION`에 있으며 `apiVersion: $GROUP_NAME/$VERSION`을 사용한다 + (예: `apiVersion: batch/v1`). 지원되는 API 그룹 전체의 목록은 [Kubernetes API reference](/docs/reference/)에서 확인할 수 있다. + + +[Custom resources](/docs/concepts/api-extension/custom-resources/)로 API를 확장하는 경우에는 두 종류의 경로가 지원된다. + +1. [CustomResourceDefinition](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/)은 아주 기본적인 + CRUD 요구를 갖는 사용자에게 적합하다. +1. 쿠버네티스 API 의미론의 전체 셋을 가지고 사용자만의 apiserver를 만들고자하는 사용자는 [aggregator](/docs/tasks/access-kubernetes-api/configure-aggregation-layer/)를 사용해서 클라이언트 입장에서 매끄럽게 동작하도록 + 만들 수 있다. + + +## API 그룹 활성화 시키기 + +특정 리소스와 API 그룹은 기본적으로 활성화되어 있다. 이들은 apiserver에서 `--runtime-config`를 설정해서 활성화하거나 +비활성화 시킬 수 있다. `--runtime-config`는 쉼표로 분리된 값을 허용한다. 예를 들어서 batch/v1을 비활성화 시키려면 +`--runtime-config=batch/v1=false`와 같이 설정하고, batch/v2alpha1을 활성화 시키려면 `--runtime-config=batch/v2alpha1`을 +설정한다. 이 플래그는 apiserver의 런타임 설정에 쉼표로 분리된 키=값 쌍의 집합을 허용한다. + +중요: 그룹이나 리소스를 활성화 또는 비활성화 시키기 위해서는 apiserver와 controller-manager를 재시작해서 +`--runtime-config` 변경을 반영시켜야 한다. + +## 그룹 내 리소스 활성화 시키기 + +데몬셋, 디플로이먼트, HorizontalPodAutoscaler, 인그레스, 잡 및 레플리카셋이 기본적으로 활성화되어 있다. +다른 확장 리소스는 apiserver의 `--runtime-config`를 설정해서 활성화 시킬 수 있다. +`--runtime-config`는 쉼표로 분리된 값을 허용한다. 예를 들어 디플로이먼트와 인그레스를 비활성화 시키려면, +`--runtime-config=extensions/v1beta1/deployments=false,extensions/v1beta1/ingress=false`와 같이 설정한다. + +{{% /capture %}} diff --git a/content/ko/docs/concepts/overview/what-is-kubernetes.md b/content/ko/docs/concepts/overview/what-is-kubernetes.md index f086ab21f5..332af17e9e 100644 --- a/content/ko/docs/concepts/overview/what-is-kubernetes.md +++ b/content/ko/docs/concepts/overview/what-is-kubernetes.md @@ -1,207 +1,178 @@ --- -reviewers: -- bgrant0607 -- mikedanese -title: What is Kubernetes? +title: 쿠버네티스란 무엇인가? content_template: templates/concept weight: 10 --- {{% capture overview %}} -This page is an overview of Kubernetes. +이 페이지에서는 쿠버네티스 개요를 설명한다. {{% /capture %}} {{% capture body %}} -Kubernetes is a portable, extensible open-source platform for managing -containerized workloads and services, that facilitates both -declarative configuration and automation. It has a large, rapidly -growing ecosystem. Kubernetes services, support, and tools are widely available. +쿠버네티스는 컨테이너화된 워크로드와 서비스를 관리하기 위한 이식성이 있고, +확장가능한 오픈소스 플랫폼이다. 쿠버네티스는 선언적 구성과 자동화를 모두 +용이하게 해준다. 쿠버네티스는 크고, 빠르게 성장하는 생태계를 가지고 있다. +쿠버네티스 서비스, 기술 지원 및 도구는 어디서나 쉽게 이용할 수 있다. -Google open-sourced the Kubernetes project in 2014. Kubernetes builds upon -a [decade and a half of experience that Google has with running -production workloads at -scale](https://research.google.com/pubs/pub43438.html), combined with -best-of-breed ideas and practices from the community. +구글이 2014년에 쿠버네티스 프로젝트를 오픈소스화했다. 쿠버네티스는 [구글의 +15여년에 걸친 대규모 운영 워크로드 운영 +경험](https://research.google.com/pubs/pub43438.html)을 기반으로 만들어졌으며 +커뮤니티의 최고의 아이디어와 적용 사례가 결합되었다. -## Why do I need Kubernetes and what can it do? +## 쿠버네티스가 왜 필요하고 무엇을 할 수 있는가? -Kubernetes has a number of features. It can be thought of as: +쿠버네티스에는 많은 기능이 있다. 다음과 같이 생각해 볼 수 있다. -- a container platform -- a microservices platform -- a portable cloud platform -and a lot more. +- 컨테이너 플랫폼 +- 마이크로서비스 플랫폼 +- 이식성 있는 클라우드 플랫폼 +그리고 더 많은 기능. -Kubernetes provides a **container-centric** management environment. It -orchestrates computing, networking, and storage infrastructure on -behalf of user workloads. This provides much of the simplicity of -Platform as a Service (PaaS) with the flexibility of Infrastructure as -a Service (IaaS), and enables portability across infrastructure -providers. +쿠버네티스는 **컨테이너 중심의** 관리 환경을 제공한다. 이 환경은 사용자 +워크로드를 위해서 컴퓨팅, 네트워킹 및 스토리지 인프라스트럭처를 +오케스트레이션한다. 이는 Platform as a Service(PaaS)의 매우 단순명료함에 +Infrastructure as a Service (IaaS)의 유연함을 더해 주며, 인프라스트럭처 +제공자 간 이식이 가능하게 해준다. -## How is Kubernetes a platform? +## 어떻게 쿠버네티스가 플랫폼이 될 수 있는가? -Even though Kubernetes provides a lot of functionality, there are -always new scenarios that would benefit from new -features. Application-specific workflows can be streamlined to -accelerate developer velocity. Ad hoc orchestration that is acceptable -initially often requires robust automation at scale. This is why -Kubernetes was also designed to serve as a platform for building an -ecosystem of components and tools to make it easier to deploy, scale, -and manage applications. +쿠버네티스가 제공하는 많은 기능이 있지만, 신규 기능을 통해 혜택을 얻을 수 있는 +새로운 시나리오는 항상 있게 마련이다. 개발자의 생산성을 극대화할 수 있도록 +애플리케이션에 특화된 워크플로우를 최적화할 수 있다. 초기에 수용 가능한 애드혹 +오케스트레이션은 대규모의 견고한 자동화를 필요로 하곤 한다. 이것이 쿠버네티스가 +애플리케이션을 더 쉽게 배포하고, 스케일링하며, 관리하는 컴포넌트와 툴의 생태계를 +만드는 플랫폼의 기능을 하도록 설계된 이유이다. -[Labels](/docs/concepts/overview/working-with-objects/labels/) empower -users to organize their resources however they -please. [Annotations](/docs/concepts/overview/working-with-objects/annotations/) -enable users to decorate resources with custom information to -facilitate their workflows and provide an easy way for management -tools to checkpoint state. +[레이블](/docs/concepts/overview/working-with-objects/labels/)은 사용자가 원하는 +방식대로 자원을 정리할 수 있도록 해준다. +[어노테이션](/docs/concepts/overview/working-with-objects/annotations/)은 +자원에 사용자 정의 정보를 추가해서 사용자의 워크플로우에 활용할 수 있도록 하고 +관리 툴이 상태를 쉽게 체크할 수 있는 방법을 제공해 준다. -Additionally, the [Kubernetes control -plane](/docs/concepts/overview/components/) is built upon the same -[APIs](/docs/reference/using-api/api-overview/) that are available to developers -and users. Users can write their own controllers, such as -[schedulers](https://github.com/kubernetes/community/blob/{{< param "githubbranch" >}}/contributors/devel/scheduler.md), -with [their own -APIs](/docs/concepts/api-extension/custom-resources/) -that can be targeted by a general-purpose [command-line -tool](/docs/user-guide/kubectl-overview/). +추가로, [쿠버네티스 컨트롤 플레인](/docs/concepts/overview/components/)은 +개발자와 사용자가 공통으로 사용할 수 있는 [API](/docs/reference/using-api/api-overview/)를 +기반으로 하고 있다. 사용자는 범용의 [명령줄 도구]((/docs/user-guide/kubectl-overview/))를 +대상으로 하는 [자체 API](/docs/concepts/api-extension/custom-resources/)를 가진 +[스케줄러](https://github.com/kubernetes/community/blob/{{< param "githubbranch" >}}/contributors/devel/scheduler.md)와 +같은 사용자만의 컨트롤러를 작성할 수 있다. -This -[design](https://git.k8s.io/community/contributors/design-proposals/architecture/architecture.md) -has enabled a number of other systems to build atop Kubernetes. +이 [설계](https://git.k8s.io/community/contributors/design-proposals/architecture/architecture.md)를 통해 +쿠버네티스 위에 많은 다른 시스템을 올릴 수 있게 된다. -## What Kubernetes is not +## 쿠버네티스가 아닌 것 -Kubernetes is not a traditional, all-inclusive PaaS (Platform as a -Service) system. Since Kubernetes operates at the container level -rather than at the hardware level, it provides some generally -applicable features common to PaaS offerings, such as deployment, -scaling, load balancing, logging, and monitoring. However, Kubernetes -is not monolithic, and these default solutions are optional and -pluggable. Kubernetes provides the building blocks for building developer -platforms, but preserves user choice and flexibility where it is -important. +쿠버네티스는 전통적인, 모든 것이 포함된 Platform as a Service(PaaS)가 +아니다. 쿠버네티스는 하드웨어 수준보다는 컨테이너 수준에서 운영되기 때문에, +PaaS가 일반적으로 제공하는 배포, 스케일링, 로드 밸런싱, 로깅 및 모니터링과 +같은 기능에서 공통점이 있기도 하다. 하지만, 쿠버네티스는 모놀리식(monolithic)하지 +않아서, 이런 기본 솔루션이 선택적이며 추가나 제거가 용이하다. 쿠버네티스는 +개발자 플랫폼을 만드는 구성 요소를 제공하지만, 필요한 경우 사용자의 선택권과 +유연성을 지켜준다. -Kubernetes: +쿠버네티스는 -* Does not limit the types of applications supported. Kubernetes aims - to support an extremely diverse variety of workloads, including - stateless, stateful, and data-processing workloads. If an - application can run in a container, it should run great on - Kubernetes. -* Does not deploy source code and does not build your - application. Continuous Integration, Delivery, and Deployment - (CI/CD) workflows are determined by organization cultures and preferences - as well as technical requirements. -* Does not provide application-level services, such as middleware - (e.g., message buses), data-processing frameworks (for example, - Spark), databases (e.g., mysql), caches, nor cluster storage systems (e.g., - Ceph) as built-in services. Such components can run on Kubernetes, and/or - can be accessed by applications running on Kubernetes through portable - mechanisms, such as the Open Service Broker. -* Does not dictate logging, monitoring, or alerting solutions. It provides - some integrations as proof of concept, and mechanisms to collect and - export metrics. -* Does not provide nor mandate a configuration language/system (e.g., - [jsonnet](https://github.com/google/jsonnet)). It provides a declarative - API that may be targeted by arbitrary forms of declarative specifications. -* Does not provide nor adopt any comprehensive machine configuration, - maintenance, management, or self-healing systems. +* 지원하는 애플리케이션의 유형을 제약하지 않는다. 쿠버네티스는 + 상태 유지가 필요 없는(stateless) 워크로드, 상태 유지가 필요한(stateful) 워크로드, + 데이터 처리를 위한 워크로드를 포함해서 극단적으로 다양한 워크로드를 지원하는 + 것을 목표로 한다. 애플리케이션이 컨테이너에서 구동될 수 있다면, 쿠버네티스에서 + 매우 잘 동작할 것이다. +* 소스 코드를 배포하지 않으며 애플리케이션을 빌드하지 않는다. + 지속적인 통합, 지속적인 배포(Delivery, Deployment)(CI/CD) 워크플로우는 + 기술적인 요구사항은 물론 조직 문화와 취향에 따라 결정된다. +* 미들웨어(예, 메시지 버스), 데이터 처리 프레임워크(예, Spark), 데이터베이스(예, mysql), + 캐시 또는 클러스터 스토리지 시스템(예, Ceph)와 같은 애플리케이션 레벨의 서비스를 + 제공하지 않는다. 이런 컴포넌트는 쿠버네티스 상에서 구동될 수 있고, 쿠버네티스 상에서 + 구동 중인 애플리케이션이 Open Service Broker와 같은 이식 가능한 메커니즘을 통해 접근할 + 수도 있다. +* 로깅, 모니터링 또는 경보 솔루션을 포함하지 않는다. 개념 증명을 위한 일부 통합이나, + 메트릭을 수집하고 노출하는 메커니즘을 제공한다. +* 기본 설정 언어/시스템(예, [jsonnet](https://github.com/google/jsonnet))을 제공하거나 + 요구하지 않는다. 선언적 명세의 임의적인 형식을 목적으로 하는 선언적 API를 제공한다. +* 포괄적인 머신 설정, 유지보수, 관리, 자동 복구 시스템을 제공하거나 채택하지 않는다. -Additionally, Kubernetes is not a mere *orchestration system*. In -fact, it eliminates the need for orchestration. The technical -definition of *orchestration* is execution of a defined workflow: -first do A, then B, then C. In contrast, Kubernetes is comprised of a -set of independent, composable control processes that continuously -drive the current state towards the provided desired state. It -shouldn't matter how you get from A to C. Centralized control is also -not required. This results in a system that is easier to use and more -powerful, robust, resilient, and extensible. +추가로, 쿠버네티스는 단순한 *오케스트레이션 시스템*이 아니다. 사실, +쿠버네티스는 오케스트레이션의 필요성을 없애준다. *오케스트레이션*의 +기술적인 정의는 A를 먼저 한 다음, B를 하고, C를 하는 것과 같이 정의된 워크플로우를 +수행하는 것이다. 반면에, 쿠버네티스는 독립적이고 조합 가능한 제어 프로세스들로 구성되어 +있다. 이 프로세스는 지속적으로 현재 상태를 입력받은 의도된 상태로 나아가도록 한다. +A에서 C로 어떻게 갔는지는 상관이 없다. 중앙화된 제어도 필요치 않다. 이로써 시스템이 +보다 더 사용하기 쉬워지고, 강력해지며, 견고하고, 회복력을 갖추게 되며, 확장 가능해진다. -## Why containers? +## 왜 컨테이너인가? -Looking for reasons why you should be using containers? +왜 컨테이너를 써야하는지 이유를 알고 싶은가? ![Why Containers?](/images/docs/why_containers.svg) -The *Old Way* to deploy applications was to install the applications -on a host using the operating-system package manager. This had the -disadvantage of entangling the applications' executables, -configuration, libraries, and lifecycles with each other and with the -host OS. One could build immutable virtual-machine images in order to -achieve predictable rollouts and rollbacks, but VMs are heavyweight -and non-portable. +애플리케이션을 배포하는 *구식의 방법*은 운영 체제의 패키지 관리자를 +사용해서 애플리케이션을 호스트에 설치하는 것이었다. 이 방식은 +애플리케이션의 실행 파일, 설정, 라이브러리 서로 간의 라이프사이클과 +호스트 OS와 얽히게 된다는 단점이 있다. 예측 가능한 롤아웃과 롤백을 +위해서 불변의 가상 머신 이미지를 만들 수도 있지만, VM은 너무 크고 +이식 가능하지 않다. -The *New Way* is to deploy containers based on operating-system-level -virtualization rather than hardware virtualization. These containers -are isolated from each other and from the host: they have their own -filesystems, they can't see each others' processes, and their -computational resource usage can be bounded. They are easier to build -than VMs, and because they are decoupled from the underlying -infrastructure and from the host filesystem, they are portable across -clouds and OS distributions. +*새로운 방법*은 하드웨어 가상화가 아닌 운영 체제 수준의 가상화에 기반한 +컨테이너를 배포하는 것이다. 이 컨테이너는 서로 격리되고 호스트와도 +격리된다. 컨테이너는 컨테이너 자체의 파일시스템을 갖고, 다른 컨테이너의 +프로세스를 알 수 없으며, 연산에 필요한 자원을 제한할 수 있다. VM보다 +빌드하기 쉬우며, 기반이 되는 인프라스트럭처와 호스트 파일시스템에서 +디커플되었기(decoupled) 때문에 클라우드나 OS 배포판 간 이식성이 있다. -Because containers are small and fast, one application can be packed -in each container image. This one-to-one application-to-image -relationship unlocks the full benefits of containers. With containers, -immutable container images can be created at build/release time rather -than deployment time, since each application doesn't need to be -composed with the rest of the application stack, nor married to the -production infrastructure environment. Generating container images at -build/release time enables a consistent environment to be carried from -development into production. Similarly, containers are vastly more -transparent than VMs, which facilitates monitoring and -management. This is especially true when the containers' process -lifecycles are managed by the infrastructure rather than hidden by a -process supervisor inside the container. Finally, with a single -application per container, managing the containers becomes tantamount -to managing deployment of the application. +컨테이너는 작고 빠르기 때문에, 애플리케이션 각각을 컨테이너 이미지로 +패키지할 수 있다. 이렇게 애플리케이션과 이미지를 일대일 관계를 갖도록 +하면 컨테이너의 혜택을 만끽할 수 있게 된다. 불변의 컨테이너 이미지는 +배포 시점이 아닌 빌드/릴리스 시점에 만들어질 수 있다. 왜냐하면 각각의 +애플리케이션은 애플리케이션 스택 외의 나머지 요소와 조합될 필요가 없기 +때문이고, 운영 인프라스트럭처 환경에 밀접하게 결합시킬 필요도 없기 +때문이다. 컨테이너 이미지를 빌드/릴리스 시점에 생성하게 되면 개발 +환경부터 운영 환경까지 일관된 환경을 가져갈 수 있게 된다. 마찬가지로, +컨테이너는 VM보다 훨씬 더 투명해서 모니터링과 관리가 용이하다. +컨테이너의 프로세스 라이프사이클이 수퍼바이저 프로세스에 의해 컨테이너 +내에 감추어지지 않고, 인프라스트럭처에 의해 관리될 때 더욱 이는 +용이해진다. 컨테이너마다 단일 애플리케이션을 담게되면, 궁극적으로 +컨테이너를 관리하는 것이 애플리케이션의 배포를 관리하는 것과 같아진다. -Summary of container benefits: +컨테이너의 혜택 요약: -* **Agile application creation and deployment**: - Increased ease and efficiency of container image creation compared to VM image use. -* **Continuous development, integration, and deployment**: - Provides for reliable and frequent container image build and - deployment with quick and easy rollbacks (due to image - immutability). -* **Dev and Ops separation of concerns**: - Create application container images at build/release time rather - than deployment time, thereby decoupling applications from - infrastructure. -* **Observability** - Not only surfaces OS-level information and metrics, but also application - health and other signals. -* **Environmental consistency across development, testing, and production**: - Runs the same on a laptop as it does in the cloud. -* **Cloud and OS distribution portability**: - Runs on Ubuntu, RHEL, CoreOS, on-prem, Google Kubernetes Engine, and anywhere else. -* **Application-centric management**: - Raises the level of abstraction from running an OS on virtual - hardware to running an application on an OS using logical resources. -* **Loosely coupled, distributed, elastic, liberated [micro-services](https://martinfowler.com/articles/microservices.html)**: - Applications are broken into smaller, independent pieces and can - be deployed and managed dynamically -- not a fat monolithic stack - running on one big single-purpose machine. -* **Resource isolation**: - Predictable application performance. -* **Resource utilization**: - High efficiency and density. +* **기민한 애플리케이션 생성과 배포**: + VM 이미지 사용 대비 컨테이너 이미지 생성이 보다 쉽고 효율적임. +* **지속적인 개발, 통합 및 배포**: + 안정적이고 주기적으로 컨테이너 이미지를 빌드해서 배포할 수 있고 + (이미지의 불변성 덕에) 빠르고 쉽게 롤백할 수 있다. +* **개발과 운영의 관심사 분리**: + 배포 시점이 아닌 빌드/릴리스 시점에 애플리케이션 컨테이너 이미지를 + 만들기 때문에, 애플리케이션이 인프라스트럭처에서 디커플된다. +* **가시성** + OS 수준의 정보와 메트릭에 머무르지 않고, 애플리케이션의 헬스와 + 그 밖의 시그널을 볼 수 있다. +* **개발, 테스팅 및 운영 환경을 걸친 일관성**: + 랩탑에서도 클라우드에서와 동일하게 구동된다. +* **클라우드 및 OS 배포판 간 이식성**: + Ubuntu, RHEL, CoreOS, on-prem, Google Kubernetes Engine 및 다른 어디에서든 구동된다. +* **애플리케이션 중심 관리**: + 가상 하드웨어의 OS에서 애플리케이션을 구동하는 수준에서 OS의 + 논리적인 자원을 사용하여 애플리케이션을 구동하는 수준으로 추상화 + 수준이 높아진다. +* **느슨하게 커플되고, 분산되고, 유연하며, 자유로운 [마이크로서비스](https://martinfowler.com/articles/microservices.html)**: + 애플리케이션은 단일 목적의 머신에서 비대한 모놀리식 스택으로 + 구동되지 않고 보다 작고 독립적인 단위로 쪼개져서 동적으로 배포되고 + 관리될 수 있다. +* **자원 격리**: + 애플리케이션 성능을 예측할 수 있다. +* **지원 사용량**: + 고효율 고집적. -## What does Kubernetes mean? K8s? +## 쿠버네티스(Kubernetes)의 뜻은? K8s? -The name **Kubernetes** originates from Greek, meaning *helmsman* or -*pilot*, and is the root of *governor* and -[cybernetic](http://www.etymonline.com/index.php?term=cybernetics). *K8s* -is an abbreviation derived by replacing the 8 letters "ubernete" with -"8". +**쿠버네티스**는 *키잡이*나 *파일럿*을 뜻하는 그리스어에서 유래했으며, +이는 *governor*(통치자)와 [cybernetic(인공두뇌학)](http://www.etymonline.com/index.php?term=cybernetics)의 +어원이다. *K8s*는 "ubernete" 8 글자를 "8"로 대체한 약어이다. {{% /capture %}} {{% capture whatsnext %}} -* Ready to [Get Started](/docs/setup/)? -* For more details, see the [Kubernetes Documentation](/docs/home/). +* [시작할](/docs/setup/) 준비가 되었는가? +* 보다 자세한 내용은 [쿠버네티스 문서](/ko/docs/home/)를 참조한다. {{% /capture %}} diff --git a/content/ko/docs/concepts/overview/working-with-objects/_index.md b/content/ko/docs/concepts/overview/working-with-objects/_index.md new file mode 100644 index 0000000000..a27acb856c --- /dev/null +++ b/content/ko/docs/concepts/overview/working-with-objects/_index.md @@ -0,0 +1,4 @@ +--- +title: "쿠버네티스 오브젝트로 작업하기" +weight: 40 +--- diff --git a/content/ko/docs/concepts/overview/working-with-objects/kubernetes-objects.md b/content/ko/docs/concepts/overview/working-with-objects/kubernetes-objects.md new file mode 100644 index 0000000000..12b6f0fa18 --- /dev/null +++ b/content/ko/docs/concepts/overview/working-with-objects/kubernetes-objects.md @@ -0,0 +1,69 @@ +--- +title: 쿠버네티스 오브젝트 이해하기 +content_template: templates/concept +weight: 10 +--- + +{{% capture overview %}} +이 페이지에서는 쿠버네티스 오브젝트가 쿠버네티스 API에서 어떻게 표현되고, 그 오브젝트를 어떻게 `.yaml` 형식으로 표현할 수 있는지에 대해 설명한다. +{{% /capture %}} + +{{% capture body %}} +## 쿠버네티스 오브젝트 이해하기 + +*쿠버네티스 오브젝트* 는 쿠버네티스 시스템에서 영속성을 가지는 개체이다. 쿠버네티스는 클러스터의 상태를 나타내기 위해 이 개체를 이용한다. 구체적으로 말하자면, 다음을 기술할 수 있다. + +* 어떤 컨테이너화된 애플리케이션이 동작 중인지 (그리고 어느 노드에서 동작 중인지) +* 그 애플리케이션이 이용할 수 있는 리소스 +* 그 애플리케이션이 어떻게 재구동 정책, 업그레이드, 그리고 내고장성과 같은 것에 동작해야 하는지에 대한 정책 + +쿠버네티스 오브젝트는 하나의 "의도를 담은 레코드" 이다. 오브젝트를 생성하게 되면, 쿠버네티스 시스템은 그 오브젝트 생성을 보장하기 위해 지속적으로 작동할 것이다. 오브젝트를 생성함으로써, 여러분이 클러스터의 워크로드를 어떤 형태로 보이고자 하는지에 대해 효과적으로 쿠버네티스 시스템에 전한다. 이것이 바로 여러분의 클러스터에 대해 **의도한 상태** 가 된다. + +생성이든, 수정이든, 또는 삭제든 쿠버네티스 오브젝트를 동작시키려면, [Kubernetes API](/docs/concepts/overview/kubernetes-api/)를 이용해야 한다. 예를 들어, `kubectl` 커맨드-라인 인터페이스를 이용할 때, CLI는 여러분 대신 필요한 쿠버네티스 API를 호출해 준다. 또한, 여러분은 [Client Libraries](/docs/reference/using-api/client-libraries/) 중 하나를 이용하여 여러분만의 프로그램에서 쿠버네티스 API를 직접 이용할 수도 있다. + +### 오브젝트 (spec)과 상태(status) + +모든 쿠버네티스 오브젝트는 오브젝트의 구성을 결정해주는 두 개의 중첩된 오브젝트 필드를 포함하는데 오브젝트 *spec* 과 오브젝트 *status* 가 그것이다. 필히 제공되어야만 하는 *spec* 은, 여러분이 오브젝트가 가졌으면 하고 원하는 특징, 즉 *의도한 상태* 를 기술한다. *status* 는 오브젝트의 *실제 상태* 를 기술하고, 쿠버네티스 시스템에 의해 제공되고 업데이트 된다. 주어진 임의의 시간에, 쿠버네티스 컨트롤 플레인은 오브젝트의 실제 상태를 여러분이 제시한 의도한 상태에 일치시키기 위해 능동적으로 관리한다. + + +예를 들어, 쿠버네티스 디플로이먼트는 클러스터에서 동작하는 애플리케이션을 표현해 줄 수 있는 오브젝트이다. 디플로이먼트를 생성할 때, 디플로이먼트 spec에 3개의 애플리케이션 레플리카가 동작되도록 설정할 수 있다. 쿠버네티스 시스템은 그 디플로이먼트 spec을 읽어 spec에 일치되도록 상태를 업데이트하여 3개의 의도한 애플리케이션 인스턴스를 구동시킨다. 만약, 그 인스턴스들 중 어느 하나가 (상태 변경에) 실패가 난다면, 쿠버네티스 시스템은 보정을 통해, 이 경우에는 인스턴스 대체를 착수하여, spec과 status 간의 차이에 대응한다. + +오브젝트 spec, staus, 그리고 metadata에 대한 추가 정보는, [Kubernetes API Conventions](https://git.k8s.io/community/contributors/devel/api-conventions.md) 를 참조한다. + +### 쿠버네티스 오브젝트 기술하기 + +쿠버네티스에서 오브젝트를 생성할 때, (이름과 같은)오브젝트에 대한 기본적인 정보와 더불어, 의도한 상태를 기술한 오브젝트 spec을 제시해 줘야만 한다. 오브젝트를 생성하기 위해(직접이든 또는 `kubectl`을 통해서든) 쿠버네티스 API를 이용할 때, API 요청은 요청 내용 안에 JSON 형식으로 정보를 포함시켜 줘야만 한다. **가장 자주, .yaml 파일로 `kubectl`에 정보를 제공해준다.** `kubectl` 은 API 요청이 이루어질 때, JSON 형식으로 정보를 변환시켜 준다. + +여기 쿠버네티스 디플로이먼트를 위한 요청 필드와 오브젝트 spec을 보여주는 `.yaml` 파일 예시가 있다. + +{{< codenew file="application/deployment.yaml" >}} + +위 예시와 같이 .yaml 파일을 이용하여 디플로이먼트를 생성하기 위한 하나의 방식으로는 `kubectl` 커맨드-라인 인터페이스에 인자값으로 `.yaml` 파일를 건네 [`kubectl create`](/docs/reference/generated/kubectl/kubectl-commands#create) 명령을 이용하는 것이다. 다음 예시와 같다. + + +```shell +$ kubectl create -f https://k8s.io/examples/application/deployment.yaml --record +``` + +그 출력 내용은 다음과 유사하다. + + +```shell +deployment.apps/nginx-deployment created +``` + +### 요구되는 필드 + +생성하고자 하는 쿠버네티스 오브젝트에 대한 `.yaml` 파일 내, 다음 필드를 위한 값들을 설정해 줘야한다. + +* `apiVersion` - 이 오브젝트를 생성하기 위해 사용하고 있는 쿠버네티스 API 버전이 어떤 것인지 +* `kind` - 어떤 종류의 오브젝트를 생성하고자 하는지 +* `metadata` - `이름` 문자열, UID, 그리고 선택적인 `네임스페이스` 를 포함하여 오브젝트를 유일하게 구분지어 줄 데이터 + +여러분은 또한 오브젝트 `spec` 필드를 제공해야만 할 것이다. 오브젝트 `spec`에 대한 정확한 포맷은 모든 쿠버네티스 오브젝트마다 다르고, 그 오브젝트 특유의 중첩된 필드를 포함한다. [Kubernetes API Reference](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/) 는 쿠버네티스를 이용하여 생성할 수 있는 오브젝트에 대한 모든 spec 포맷을 살펴볼 수 있도록 해준다. 예를 들어, `파드` 오브젝트에 대한 `spec` 포맷은 [여기](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podspec-v1-core)에서 확인할 수 있고, `디플로이먼트` 오브젝트에 대한 `spec` 포맷은 [여기](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#deploymentspec-v1-apps)에서 확인할 수 있다. + +{{% /capture %}} + +{{% capture whatsnext %}} +* [Pod](/docs/concepts/workloads/pods/pod-overview/)와 같이, 가장 중요하고 기본적인 쿠버네티스 오브젝트에 대해 배운다. +{{% /capture %}} diff --git a/content/ko/docs/concepts/workloads/_index.md b/content/ko/docs/concepts/workloads/_index.md new file mode 100644 index 0000000000..bdb9770e5d --- /dev/null +++ b/content/ko/docs/concepts/workloads/_index.md @@ -0,0 +1,4 @@ +--- +title: "워크로드" +weight: 60 +--- diff --git a/content/ko/docs/concepts/workloads/pods/_index.md b/content/ko/docs/concepts/workloads/pods/_index.md new file mode 100644 index 0000000000..35ded5ad4a --- /dev/null +++ b/content/ko/docs/concepts/workloads/pods/_index.md @@ -0,0 +1,4 @@ +--- +title: "파드" +weight: 10 +--- diff --git a/content/ko/docs/concepts/workloads/pods/pod-overview.md b/content/ko/docs/concepts/workloads/pods/pod-overview.md new file mode 100644 index 0000000000..06ace4c72d --- /dev/null +++ b/content/ko/docs/concepts/workloads/pods/pod-overview.md @@ -0,0 +1,107 @@ +--- +title: 파드(Pod) 개요 +content_template: templates/concept +weight: 10 +--- + +{{% capture overview %}} +이 페이지는 쿠버네티스 객체 모델 중 가장 작은 배포 가능한 객체인 `파드` 에 대한 개요를 제공한다. +{{% /capture %}} + + +{{% capture body %}} +## 파드에 대해 이해하기 + +*파드* 는 쿠버네티스의 기본 구성 요소이다. 쿠버네티스 객체 모델 중 만들고 배포할 수 있는 가장 작고 간단한 단위이다. 파드는 클러스터에서의 Running 프로세스를 나타낸다. + +파드는 애플리케이션 컨테이너(또는, 몇몇의 경우, 다중 컨테이너), 저장소 리소스, 특정 네트워크 IP 그리고, 컨테이너가 동작하기 위해 만들어진 옵션들을 캡슐화 한다. +파드는 배포의 단위를 말한다. 아마 단일 컨테이너로 구성되어 있거나, 강하게 결합되어 리소스를 공유하는 소수의 컨테이너로 구성되어 있는 *쿠버네티스에서의 애플리케이션 단일 인스턴스* 를 의미함. + +> [Docker](https://www.docker.com)는 쿠버네티스 파드에서 사용되는 가장 대표적인 컨테이너 런타임이지만, 파드는 다른 컨테이너 런타임 역시 지원한다. + + +쿠버네티스 클러스터 내부의 파드는 주로 두 가지 방법으로 사용된다. + +* **단일 컨테이너만 동작하는 파드**. "단일 컨테이너 당 한 개의 파드" 모델은 쿠버네티스 사용 사례 중 가장 흔하다. 이 경우, 한 개의 파드가 단일 컨테이너를 감싸고 있다고 생각할 수 있으며, 쿠버네티스는 컨테이너가 아닌 파드를 직접 관리한다고 볼 수 있다. + +* **함께 동작하는 작업이 필요한 다중 컨테이너가 동작하는 파드**. 아마 파드는 강하게 결합되어 있고 리소스 공유가 필요한 다중으로 함께 배치된 컨테이너로 구성되어 있을 것이다. 이렇게 함께 배치되어 설치된 컨테이너는 단일 결합 서비스 단위일 것이다. 한 컨테이너는 공유 볼륨에서 퍼블릭으로 파일들을 옮기고, 동시에 분리되어 있는 "사이드카" 컨테이너는 그 파일들을 업데이트 하거나 복구한다. 파드는 이 컨테이너와 저장소 리소스들을 한 개의 관리 가능한 요소로 묶는다. + + +[쿠버네티스 블로그](http://blog.kubernetes.io)에는 파드 사용 사례의 몇 가지 추가적인 정보가 있다. 더 많은 정보를 위해서 아래 내용을 참조하길 바란다. + +* [분산 시스템 툴킷: 복합 컨테이너를 위한 패턴](https://kubernetes.io/blog/2015/06/the-distributed-system-toolkit-patterns) +* [컨테이너 디자인 패턴](https://kubernetes.io/blog/2016/06/container-design-patterns) + +각각의 파드는 주어진 애플리케이션에서 단일 인스턴스로 동작을 하는 것을 말한다. 만약 애플리케이션을 수평적으로 스케일하기를 원하면(예를 들면, 다중 인스턴스 동작하는 것), 각 인스턴스 당 한 개씩 다중 파드를 사용해야 한다. 쿠버네티스에서는, 일반적으로 이것을 _복제_ 라고 한다. 복제된 파드는 주로 컨트롤러라고 하는 추상화 개념의 그룹에 의해 만들어지고 관리된다. 더 많은 정보는 [파드와 컨트롤러](#pods-and-controllers)를 참고하길 바란다. + + + +## 어떻게 파드가 다중 컨테이너를 관리하는가 + +파드는 결합도가 있는 단위의 서비스를 형성하는 다중 협력 프로세스(컨테이너)를 지원하도록 디자인 되었다. 파드 내부의 컨테이너는 자동으로 동일한 물리적 또는 가상의 머신의 클러스터에 함께 배치되고 스케쥴된다. 컨테이너는 리소스와 의존성 공유, 다른 컨테이너와의 통신 그리고 언제,어떻게 조절하는지를 공유할 수 있다. + +단일 파드 내부에서 함께 배치되고 관리되는 컨테이너 그룹은 상대적으로 심화된 사용 예시임에 유의하자. 컨테이너가 강하게 결합된 특별한 인스턴스의 경우에만 이 패턴을 사용하는게 좋다. 예를 들어, 공유 볼륨 내부 파일의 웹 서버 역할을 하는 컨테이너와 원격 소스로부터 그 파일들을 업데이트하는 분리된 "사이드카" 컨테이너가 있는 경우 아래 다이어그램의 모습일 것이다. + + +{{< figure src="/images/docs/pod.svg" title="pod diagram" width="50%" >}} + +파드는 같은 파드 안에 속한 컨테이너에게 두 가지 공유 리소스를 제공한다. *네트워킹* 과 *저장소*. + +#### 네트워킹 + +각각의 파드는 유일한 IP주소를 할당 받는다. 한 파드 내부의 모든 컨테이너는 네트워크 네임스페이스와 IP주소 및 네트워크 포트를 공유한다. *파드 안에 있는* 컨테이너는 다른 컨테이너와 `localhost`를 통해서 통신할 수 있다. 특정 파드 안에 있는 컨테이너가 *파드 밖의* 요소들과 통신하기 위해서는, 네트워크 리소스를 어떻게 쓰고 있는지 공유 해야 한다(예를 들어 포트 등). + +#### 저장소 + +파드는 공유 저장소 집합인 *볼륨* 을 명시할 수 있다. 파드 내부의 모든 컨테이너는 공유 볼륨에 접근할 수 있고, 그 컨테이너끼리 데이터를 공유하는 것을 허용한다. 또한 볼륨은 컨테이너가 재시작되어야 하는 상황에도 파드 안의 데이터가 영구적으로 유지될 수 있게 한다. 쿠버네티스가 어떻게 파드 안의 공유 저장소를 사용하는지 보려면 [볼륨](/docs/concepts/storage/volumes/)를 참고하길 바란다. + +## 파드 작업 + +직접 쿠버네티스에서 싱글톤 파드이더라도 개별 파드를 만들일이 거의 없을 것이다. 그 이유는 파드가 상대적으로 수명이 짧고 일시적이기 때문이다. 파드가 만들어지면(직접 만들거나, 컨트롤러에 의해서 간접적으로 만들어지거나), 그것은 클러스터의 노드에서 동작할 것이다. 파드는 프로세스가 종료되거나, 파드 객체가 삭제되거나, 파드가 리소스의 부족으로 인해 *제거되거나*, 노드에 장애가 생기지 않는 한 노드에 남아있는다. + +{{< note >}} +**참고:** 파드 내부에서 재시작되는 컨테이너를 파드와 함께 재시작되는 컨테이너로 혼동해서는 안된다. 파드는 자기 스스로 동작하지 않는다. 하지만 컨테이너 환경은 그것이 삭제될 때까지 계속 동작한다. +{{< /note >}} + +파드는 스스로 자신을 치료하지 않는다. 만약 파드가 스케줄링된 노드에 장애가 생기거나, 스케쥴링 동작이 스스로 실패할 경우 파드는 삭제된다. 그와 비슷하게, 파드는 리소스나 노드의 유지 부족으로 인해 제거되는 상황에서 살아남지 못할 것이다. +쿠버네티스는 상대적으로 일시적인 파드 인스턴스를 관리하는 작업을 처리하는 *컨트롤러* 라고 하는 고수준의 추상적 개념을 사용한다. 즉, 파드를 직접적으로 사용가능 하지만, 컨트롤러를 사용하여 파드를 관리하는 것이 쿠버네티스에서 훨씬 더 보편적이다. 쿠버네티스가 어떻게 파드 스케일링과 치료하는지 보려면 [파드와 컨트롤러](#pods-and-controllers)를 참고하길 바란다. + +### 파드와 컨트롤러 + +컨트롤러는 다중 파드를 생성하고 관리해 주는데, 클러스터 범위 내에서의 레플리케이션 핸들링, 롤아웃 그리고 셀프힐링 기능 제공을 한다. 예를 들어, 만약 노드가 고장났을 때, 컨트롤러는 다른 노드에 파드를 스케줄링 함으로써 자동으로 교체할 것이다. + +한 가지 또는 그 이상의 파드를 보유한 컨트롤러의 몇 가지 예시. + +* [디플로이먼트](/docs/concepts/workloads/controllers/deployment/) +* [스테이트풀 셋](/docs/concepts/workloads/controllers/statefulset/) +* [데몬 셋](/docs/concepts/workloads/controllers/daemonset/) + +일반적으로, 컨트롤러는 책임을 지고 제공한 파드 템플릿을 사용한다. + +## 파드 템플릿 +파드 템플릿은 [레플리케이션 컨트롤러](/docs/concepts/workloads/controllers/replicationcontroller/), [잡](/docs/concepts/jobs/run-to-completion-finite-workloads/), [데몬 셋](/docs/concepts/workloads/controllers/daemonset/)과 같은 다른 객체를 포함하는 파드 명세서이다. 컨트롤러는 파드 템플릿을 사용하여 실제 파드를 만든다. +아래 예시는 메시지를 출력하는 컨테이너를 포함하는 파드에 대한 간단한 매니페스트이다. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: myapp-pod + labels: + app: myapp +spec: + containers: + - name: myapp-container + image: busybox + command: ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600'] +``` + +모든 레플리카의 현재 원하는 상태를 지정하는 대신, 파드 템플릿은 쿠키 틀과 같다. 쿠키가 한 번 잘리면, 그 쿠키는 쿠키 틀과 더이상 관련이 없다. 양자 얽힘이 없는 것이다. 그 이후 템플릿을 변경하거나 새로운 템플릿으로 바꿔도 이미 만들어진 파드에는 직접적인 영향이 없다. 마찬가지로, 레플리케이션 컨트롤러에 의해 만들어진 파드는 아마 그 이후 직접 업데이트될 수 있다. 이것은 모든 컨테이너가 속해있는 파드에서 현재 원하는 상태를 명시하는 것과 의도적으로 대비가 된다. 이러한 접근은 시스템의 의미를 철저히 단순화하고 유연성을 증가시킨다. + +{{% /capture %}} + +{{% capture whatsnext %}} +* 파드의 다른 동작들을 더 배워보자. + * [파드 종료](/docs/concepts/workloads/pods/pod/#termination-of-pods) + * 다른 파드 주제 +{{% /capture %}} diff --git a/content/ko/docs/home/_index.md b/content/ko/docs/home/_index.md index da2e4ffd38..542d38e7f3 100644 --- a/content/ko/docs/home/_index.md +++ b/content/ko/docs/home/_index.md @@ -6,7 +6,7 @@ cid: userJourneys css: /css/style_user_journeys.css js: /js/user-journeys/home.js, https://use.fontawesome.com/4bcc658a89.js display_browse_numbers: true -linkTitle: "문서" +linkTitle: "홈" main_menu: true weight: 10 menu: diff --git a/content/ko/docs/reference/_index.md b/content/ko/docs/reference/_index.md new file mode 100644 index 0000000000..899f39f4f3 --- /dev/null +++ b/content/ko/docs/reference/_index.md @@ -0,0 +1,58 @@ +--- +title: 레퍼런스 +linkTitle: "레퍼런스" +main_menu: true +weight: 70 +content_template: templates/concept +--- + +{{% capture overview %}} + +쿠버네티스 문서의 본 섹션에서는 레퍼런스를 다룬다. + +{{% /capture %}} + +{{% capture body %}} + +## API 레퍼런스 + +* [Kubernetes API Overview](/docs/reference/using-api/api-overview/) - 쿠버네티스 API에 대한 개요 +* 쿠버네티스 API 버전 + * [1.12](/docs/reference/generated/kubernetes-api/v1.12/) + * [1.11](/docs/reference/generated/kubernetes-api/v1.11/) + * [1.10](https://v1-10.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/) + * [1.9](https://v1-9.docs.kubernetes.io/docs/api-reference/v1.9/) + * [1.8](https://v1-8.docs.kubernetes.io/docs/api-reference/v1.8/) + * [1.7](https://v1-7.docs.kubernetes.io/docs/api-reference/v1.7/) + +## API 클라이언트 라이브러리 + +프로그래밍 언어에서 쿠버네티스 API를 호출하기 위해서, +[client libraries](/docs/reference/using-api/client-libraries/)를 사용할 수 있다. +공식적으로 지원되는 클라이언트 라이브러리는 다음과 같다. + +- [Kubernetes Go client library](https://github.com/kubernetes/client-go/) +- [Kubernetes Python client library](https://github.com/kubernetes-client/python) + +## CLI 레퍼런스 + +* [kubectl](/docs/user-guide/kubectl-overview) - 명령어를 실행하거나 쿠버네티스 클러스터를 관리하기 위해 사용하는 주된 CLI 도구. + * [JSONPath](/docs/user-guide/jsonpath/) - kubectl에서 [JSONPath expressions](http://goessner.net/articles/JsonPath/)을 사용하기 위한 문법 가이드. +* [kubeadm](/docs/admin/kubeadm/) - 안정적인 쿠버네티스 클러스터를 쉽게 프로비전하기 위한 CLI 도구. +* [kubefed](/docs/admin/kubefed/) - 연합된(federated) 클러스터 관리를 도와주는 CLI 도구. + +## 설정 레퍼런스 + +* [kubelet](/docs/admin/kubelet/) - 각 노드에서 구동되는 주요한 *노드 에이전트*. kubelet은 PodSpecs 집합을 가지며 기술된 컨테이너가 구동되고 있는지, 정상 작동하는지를 보장한다. +* [kube-apiserver](/docs/admin/kube-apiserver/) - 파드, 서비스, 레플리케이션 컨트롤러와 같은 API 오브젝트에 대한 검증과 구성을 수행하는 REST API. +* [kube-controller-manager](/docs/admin/kube-controller-manager/) - 쿠버네티스에 탑재된 핵심 제어 루프를 포함하는 데몬. +* [kube-proxy](/docs/admin/kube-proxy/) - 간단한 TCP/UDP 스트림 포워딩이나 백-엔드 집합에 걸쳐서 라운드-로빈 TCP/UDP 포워딩을 할 수 있다. +* [kube-scheduler](/docs/admin/kube-scheduler/) - 가용성, 성능 및 용량을 관리하는 스케줄러. +* [federation-apiserver](/docs/admin/federation-apiserver/) - 연합된 클러스터를 위한 API 서버. +* [federation-controller-manager](/docs/admin/federation-controller-manager/) - 쿠버네티스 연합에 탑재된 핵심 제어 루프를 포함하는 데몬. + +## 설계 문서 + +쿠버네티스 기능에 대한 설계 문서의 아카이브. [Kubernetes Architecture](https://git.k8s.io/community/contributors/design-proposals/architecture/architecture.md)와 [Kubernetes Design Overview](https://git.k8s.io/community/contributors/design-proposals)가 좋은 출발점이다. + +{{% /capture %}} diff --git a/content/ko/docs/reference/glossary/controller.md b/content/ko/docs/reference/glossary/controller.md new file mode 100644 index 0000000000..5c810e2b47 --- /dev/null +++ b/content/ko/docs/reference/glossary/controller.md @@ -0,0 +1,19 @@ +--- +title: Controller +id: controller +date: 2018-04-12 +full_link: /docs/admin/kube-controller-manager/ +short_description: > + A control loop that watches the shared state of the cluster through the apiserver and makes changes attempting to move the current state towards the desired state. + +aka: +tags: +- architecture +- fundamental +--- + A control loop that watches the shared state of the cluster through the {{< glossary_tooltip text="apiserver" term_id="kube-apiserver" >}} and makes changes attempting to move the current state towards the desired state. + + + +Examples of controllers that ship with Kubernetes today are the replication controller, endpoints controller, namespace controller, and serviceaccounts controller. + diff --git a/content/ko/docs/reference/glossary/etcd.md b/content/ko/docs/reference/glossary/etcd.md new file mode 100644 index 0000000000..6e8d5186bc --- /dev/null +++ b/content/ko/docs/reference/glossary/etcd.md @@ -0,0 +1,19 @@ +--- +title: etcd +id: etcd +date: 2018-04-12 +full_link: /docs/tasks/administer-cluster/configure-upgrade-etcd/ +short_description: > + Consistent and highly-available key value store used as Kubernetes' backing store for all cluster data. + +aka: +tags: +- architecture +- storage +--- + Consistent and highly-available key value store used as Kubernetes' backing store for all cluster data. + + + +Always have a backup plan for etcd's data for your Kubernetes cluster. For in-depth information on etcd, see [etcd documentation](https://github.com/coreos/etcd/blob/master/Documentation/docs.md). + diff --git a/content/ko/docs/reference/glossary/kube-apiserver.md b/content/ko/docs/reference/glossary/kube-apiserver.md new file mode 100644 index 0000000000..d6855edb38 --- /dev/null +++ b/content/ko/docs/reference/glossary/kube-apiserver.md @@ -0,0 +1,19 @@ +--- +title: kube-apiserver +id: kube-apiserver +date: 2018-04-12 +full_link: /docs/reference/generated/kube-apiserver/ +short_description: > + Component on the master that exposes the Kubernetes API. It is the front-end for the Kubernetes control plane. + +aka: +tags: +- architecture +- fundamental +--- + Component on the master that exposes the Kubernetes API. It is the front-end for the Kubernetes control plane. + + + +It is designed to scale horizontally -- that is, it scales by deploying more instances. See [Building High-Availability Clusters](/docs/admin/high-availability/). + diff --git a/content/ko/docs/reference/glossary/kube-controller-manager.md b/content/ko/docs/reference/glossary/kube-controller-manager.md new file mode 100644 index 0000000000..0b039ae536 --- /dev/null +++ b/content/ko/docs/reference/glossary/kube-controller-manager.md @@ -0,0 +1,19 @@ +--- +title: kube-controller-manager +id: kube-controller-manager +date: 2018-04-12 +full_link: /docs/reference/generated/kube-controller-manager/ +short_description: > + Component on the master that runs controllers. + +aka: +tags: +- architecture +- fundamental +--- + Component on the master that runs {{< glossary_tooltip text="controllers" term_id="controller" >}}. + + + +Logically, each {{< glossary_tooltip text="controller" term_id="controller" >}} is a separate process, but to reduce complexity, they are all compiled into a single binary and run in a single process. + diff --git a/content/ko/docs/reference/glossary/kube-scheduler.md b/content/ko/docs/reference/glossary/kube-scheduler.md new file mode 100644 index 0000000000..b9672b6e03 --- /dev/null +++ b/content/ko/docs/reference/glossary/kube-scheduler.md @@ -0,0 +1,18 @@ +--- +title: kube-scheduler +id: kube-scheduler +date: 2018-04-12 +full_link: /docs/reference/generated/kube-scheduler/ +short_description: > + Component on the master that watches newly created pods that have no node assigned, and selects a node for them to run on. + +aka: +tags: +- architecture +--- + Component on the master that watches newly created pods that have no node assigned, and selects a node for them to run on. + + + +Factors taken into account for scheduling decisions include individual and collective resource requirements, hardware/software/policy constraints, affinity and anti-affinity specifications, data locality, inter-workload interference and deadlines. + diff --git a/content/ko/docs/reference/glossary/kubelet.md b/content/ko/docs/reference/glossary/kubelet.md new file mode 100644 index 0000000000..0c4ea9425a --- /dev/null +++ b/content/ko/docs/reference/glossary/kubelet.md @@ -0,0 +1,19 @@ +--- +title: Kubelet +id: kubelet +date: 2018-04-12 +full_link: /docs/reference/generated/kubelet +short_description: > + An agent that runs on each node in the cluster. It makes sure that containers are running in a pod. + +aka: +tags: +- fundamental +- core-object +--- + An agent that runs on each node in the cluster. It makes sure that containers are running in a pod. + + + +The kubelet takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy. The kubelet doesn’t manage containers which were not created by Kubernetes. + diff --git a/content/ko/docs/setup/minikube.md b/content/ko/docs/setup/minikube.md index 1507cce2da..0246b91cc6 100644 --- a/content/ko/docs/setup/minikube.md +++ b/content/ko/docs/setup/minikube.md @@ -60,11 +60,34 @@ NAME READY STATUS RESTARTS AGE hello-minikube-3383150820-vctvh 1/1 Running 0 13s # We can see that the pod is now Running and we will now be able to curl it: $ curl $(minikube service hello-minikube --url) -CLIENT VALUES: -client_address=192.168.99.1 -command=GET -real path=/ -... + + +Hostname: hello-minikube-7c77b68cff-8wdzq + +Pod Information: + -no pod information available- + +Server values: + server_version=nginx: 1.13.3 - lua: 10008 + +Request Information: + client_address=172.17.0.1 + method=GET + real path=/ + query= + request_version=1.1 + request_scheme=http + request_uri=http://192.168.99.100:8080/ + +Request Headers: + accept=*/* + host=192.168.99.100:30674 + user-agent=curl/7.47.0 + +Request Body: + -no body in request- + + $ kubectl delete services hello-minikube service "hello-minikube" deleted $ kubectl delete deployment hello-minikube @@ -191,13 +214,6 @@ To switch back to this context later, run this command: `kubectl config use-cont #### 쿠버네티스 버전 지정 -Minikube supports running multiple different versions of Kubernetes. You can -access a list of all available versions via - -``` -minikube get-k8s-versions -``` - You can specify the specific version of Kubernetes for Minikube to use by adding the `--kubernetes-version` string to the `minikube start` command. For example, to run version `v1.7.3`, you would run the following: diff --git a/content/ko/docs/setup/multiple-zones.md b/content/ko/docs/setup/multiple-zones.md index 0feb902f09..e00d6f6dba 100644 --- a/content/ko/docs/setup/multiple-zones.md +++ b/content/ko/docs/setup/multiple-zones.md @@ -54,7 +54,7 @@ There are some important limitations of the multizone support: * We assume that the different zones are located close to each other in the network, so we don't perform any zone-aware routing. In particular, traffic -that goes via services might cross zones (even if pods in some pods backing that service +that goes via services might cross zones (even if some pods backing that service exist in the same zone as the client), and this may incur additional latency and cost. * Volume zone-affinity will only work with a `PersistentVolume`, and will not @@ -122,10 +122,10 @@ and `failure-domain.beta.kubernetes.io/zone` for the zone: NAME STATUS ROLES AGE VERSION LABELS -kubernetes-master Ready,SchedulingDisabled 6m v1.11.1 beta.kubernetes.io/instance-type=n1-standard-1,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-master -kubernetes-minion-87j9 Ready 6m v1.11.1 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-87j9 -kubernetes-minion-9vlv Ready 6m v1.11.1 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-9vlv -kubernetes-minion-a12q Ready 6m v1.11.1 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-a12q +kubernetes-master Ready,SchedulingDisabled 6m v1.12.0 beta.kubernetes.io/instance-type=n1-standard-1,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-master +kubernetes-minion-87j9 Ready 6m v1.12.0 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-87j9 +kubernetes-minion-9vlv Ready 6m v1.12.0 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-9vlv +kubernetes-minion-a12q Ready 6m v1.12.0 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-a12q ``` ### 두번째 영역에 더 많은 노드 추가하기 @@ -157,13 +157,13 @@ in us-central1-b: > kubectl get nodes --show-labels NAME STATUS ROLES AGE VERSION LABELS -kubernetes-master Ready,SchedulingDisabled 16m v1.11.1 beta.kubernetes.io/instance-type=n1-standard-1,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-master -kubernetes-minion-281d Ready 2m v1.11.1 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-b,kubernetes.io/hostname=kubernetes-minion-281d -kubernetes-minion-87j9 Ready 16m v1.11.1 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-87j9 -kubernetes-minion-9vlv Ready 16m v1.11.1 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-9vlv -kubernetes-minion-a12q Ready 17m v1.11.1 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-a12q -kubernetes-minion-pp2f Ready 2m v1.11.1 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-b,kubernetes.io/hostname=kubernetes-minion-pp2f -kubernetes-minion-wf8i Ready 2m v1.11.1 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-b,kubernetes.io/hostname=kubernetes-minion-wf8i +kubernetes-master Ready,SchedulingDisabled 16m v1.12.0 beta.kubernetes.io/instance-type=n1-standard-1,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-master +kubernetes-minion-281d Ready 2m v1.12.0 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-b,kubernetes.io/hostname=kubernetes-minion-281d +kubernetes-minion-87j9 Ready 16m v1.12.0 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-87j9 +kubernetes-minion-9vlv Ready 16m v1.12.0 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-9vlv +kubernetes-minion-a12q Ready 17m v1.12.0 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-a12q +kubernetes-minion-pp2f Ready 2m v1.12.0 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-b,kubernetes.io/hostname=kubernetes-minion-pp2f +kubernetes-minion-wf8i Ready 2m v1.12.0 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-b,kubernetes.io/hostname=kubernetes-minion-wf8i ``` ### 볼륨 어피니티 @@ -286,9 +286,9 @@ Node: kubernetes-minion-olsh/10.240.0.11 > kubectl get node kubernetes-minion-9vlv kubernetes-minion-281d kubernetes-minion-olsh --show-labels NAME STATUS ROLES AGE VERSION LABELS -kubernetes-minion-9vlv Ready 34m v1.11.1 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-9vlv -kubernetes-minion-281d Ready 20m v1.11.1 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-b,kubernetes.io/hostname=kubernetes-minion-281d -kubernetes-minion-olsh Ready 3m v1.11.1 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-f,kubernetes.io/hostname=kubernetes-minion-olsh +kubernetes-minion-9vlv Ready 34m v1.12.0 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-a,kubernetes.io/hostname=kubernetes-minion-9vlv +kubernetes-minion-281d Ready 20m v1.12.0 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-b,kubernetes.io/hostname=kubernetes-minion-281d +kubernetes-minion-olsh Ready 3m v1.12.0 beta.kubernetes.io/instance-type=n1-standard-2,failure-domain.beta.kubernetes.io/region=us-central1,failure-domain.beta.kubernetes.io/zone=us-central1-f,kubernetes.io/hostname=kubernetes-minion-olsh ``` diff --git a/content/ko/docs/setup/scratch.md b/content/ko/docs/setup/scratch.md index 20edb1feb3..9732925d8c 100644 --- a/content/ko/docs/setup/scratch.md +++ b/content/ko/docs/setup/scratch.md @@ -194,7 +194,7 @@ We recommend that you use the etcd version which is provided in the Kubernetes b were tested extensively with this version of etcd and not with any other version. The recommended version number can also be found as the value of `TAG` in `kubernetes/cluster/images/etcd/Makefile`. -For the miniumum recommended version of etcd, refer to +For the minimum recommended version of etcd, refer to [Configuring and Updating etcd](/docs/tasks/administer-cluster/configure-upgrade-etcd/) The remainder of the document assumes that the image identifiers have been chosen and stored in corresponding env vars. Examples (replace with latest tags and appropriate registry): diff --git a/content/ko/docs/tasks/_index.md b/content/ko/docs/tasks/_index.md new file mode 100644 index 0000000000..6e9fc1b326 --- /dev/null +++ b/content/ko/docs/tasks/_index.md @@ -0,0 +1,87 @@ +--- +title: 태스크 +main_menu: true +weight: 50 +content_template: templates/concept +--- + +{{< toc >}} + +{{% capture overview %}} + +This section of the Kubernetes documentation contains pages that +show how to do individual tasks. A task page shows how to do a +single thing, typically by giving a short sequence of steps. + +{{% /capture %}} + +{{% capture body %}} + +## Web UI (Dashboard) + +Deploy and access the Dashboard web user interface to help you manage and monitor containerized applications in a Kubernetes cluster. + +## Using the kubectl Command-line + +Install and setup the `kubectl` command-line tool used to directly manage Kubernetes clusters. + +## Configuring Pods and Containers + +Perform common configuration tasks for Pods and Containers. + +## Running Applications + +Perform common application management tasks, such as rolling updates, injecting information into pods, and horizontal Pod autoscaling. + +## Running Jobs + +Run Jobs using parallel processing. + +## Accessing Applications in a Cluster + +Configure load balancing, port forwarding, or setup firewall or DNS configurations to access applications in a cluster. + +## Monitoring, Logging, and Debugging + +Setup monitoring and logging to troubleshoot a cluster or debug a containerized application. + +## Accessing the Kubernetes API + +Learn various methods to directly access the Kubernetes API. + +## Using TLS + +Configure your application to trust and use the cluster root Certificate Authority (CA). + +## Administering a Cluster + +Learn common tasks for administering a cluster. + +## Administering Federation + +Configure components in a cluster federation. + +## Managing Stateful Applications + +Perform common tasks for managing Stateful applications, including scaling, deleting, and debugging StatefulSets. + +## Cluster Daemons + +Perform common tasks for managing a DaemonSet, such as performing a rolling update. + +## Managing GPUs + +Configure and schedule NVIDIA GPUs for use as a resource by nodes in a cluster. + +## Managing HugePages + +Configure and schedule huge pages as a schedulable resource in a cluster. + +{{% /capture %}} + +{{% capture whatsnext %}} + +If you would like to write a task page, see +[Creating a Documentation Pull Request](/docs/home/contribute/create-pull-request/). + +{{% /capture %}} diff --git a/content/ko/docs/tasks/run-application/_index.md b/content/ko/docs/tasks/run-application/_index.md new file mode 100644 index 0000000000..203d2eb39c --- /dev/null +++ b/content/ko/docs/tasks/run-application/_index.md @@ -0,0 +1,4 @@ +--- +title: "애플리케이션 실행" +weight: 40 +--- diff --git a/content/ko/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md b/content/ko/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md new file mode 100644 index 0000000000..b52ca2dac5 --- /dev/null +++ b/content/ko/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md @@ -0,0 +1,375 @@ +--- +title: Horizontal Pod Autoscaler 연습 +content_template: templates/task +weight: 100 +--- + +{{% capture overview %}} + +Horizontal Pod Autoscaler는 CPU 사용량(또는 베타 지원의 다른 애플리케이션 지원 메트릭)을 관찰하여 레플리케이션 컨트롤러, 디플로이먼트 또는 레플리카 셋의 파드 개수를 자동으로 스케일한다. + +이 문서는 php-apache 서버를 대상으로 Horizontal Pod Autoscaler를 동작해보는 예제이다. Horizontal Pod Autoscaler 동작과 관련된 더 많은 정보를 위해서는 [Horizontal Pod Autoscaler 사용자 가이드](/docs/tasks/run-application/horizontal-pod-autoscale/)를 참고하기 바란다. + +{{% /capture %}} + + + +{{% capture prerequisites %}} + +이 예제는 버전 1.2 또는 이상의 쿠버네티스 클러스터와 kubectl을 필요로 한다. +[메트릭-서버](https://github.com/kubernetes-incubator/metrics-server/) 모니터링을 클러스터에 배포하여 리소스 메트릭 API를 통해 메트릭을 제공해야 한다. 이는 Horizontal Pod Autoscaler가 메트릭을 수집할때 해당 API를 사용한다. ([GCE 가이드](/docs/setup/turnkey/gce/)로 클러스터를 올리는 경우 메트릭-서버 모니터링은 디폴트로 활성화된다.) + +Horizontal Pod Autoscaler에 다양한 자원 메트릭을 적용하고자 하는 경우, 버전 1.6 또는 이상의 쿠버네티스 클러스터와 kubectl를 사용해야 한다. 또한, 사용자 정의 메트릭을 사용하기 위해서는, 클러스터가 사용자 정의 메트릭 API를 제공하는 API 서버와 통신할 수 있어야 한다. 마지막으로, 쿠버네티스 오브젝트와 관련이 없는 메트릭을 사용하는 경우 버전 1.10 또는 이상의 쿠버네티스 클러스터와 kubectl을 사용해야 하며, 외부 메트릭 API와 통신이 가능해야 한다. 자세한 사항은 [Horizontal Pod Autoscaler 사용자 가이드](/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-custom-metrics)를 참고하길 바란다. + +{{% /capture %}} + +{{% capture steps %}} + +## php-apache 서버 구동 및 노출 + +Horizontal Pod Autoscaler 시연을 위해 php-apache 이미지를 맞춤 제작한 Docker 이미지를 사용한다. +Dockerfile은 다음과 같다. + + +``` +FROM php:5-apache +ADD index.php /var/www/html/index.php +RUN chmod a+rx index.php +``` + +index.php는 CPU 과부하 연산을 수행한다. + +``` + +``` + +첫 번째 단계로, 실행 중인 이미지의 디플로이먼트를 시작하고 서비스로 노출시킨다. + +```shell +$ kubectl run php-apache --image=k8s.gcr.io/hpa-example --requests=cpu=200m --expose --port=80 +service/php-apache created +deployment.apps/php-apache created +``` + +## Horizontal Pod Autoscaler 생성 + +이제 서비스가 동작중이므로, [kubectl autoscale](https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/docs/user-guide/kubectl/kubectl_autoscale.md)를 사용하여 오토스케일러를 생성한다. 다음 명령어는 첫 번째 단계에서 만든 php-apache 디플로이먼트 파드의 개수를 1부터 10 사이로 유지하는 Horizontal Pod Autoscaler를 생성한다. +간략히 얘기하면, HPA는 (디플로이먼트를 통한) 평균 CPU 사용량을 50%로 유지하기 위하여 레플리카의 개수를 늘리고 줄인다. ([kubectl run](https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/docs/user-guide/kubectl/kubectl_run.md)으로 각 파드는 200 밀리코어까지 요청할 수 있고, 따라서 여기서 말하는 평균 CPU 사용은 100 밀리코어를 말한다.) 이에 대한 자세한 알고리즘은 [여기](https://git.k8s.io/community/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md#autoscaling-algorithm)를 참고하기 바란다. + +```shell +$ kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10 +horizontalpodautoscaler.autoscaling/php-apache autoscaled +``` + +실행 중인 오토스케일러의 현재 상태를 확인해본다. +```shell +$ kubectl get hpa +NAME REFERENCE TARGET MINPODS MAXPODS REPLICAS AGE +php-apache Deployment/php-apache/scale 0% / 50% 1 10 1 18s + +``` + +아직 서버로 어떠한 요청도 하지 않았기 때문에, 현재 CPU 소비는 0%임을 확인할 수 있다 (``CURRENT``은 디플로이먼트에 의해 제어되는 파드들의 평균을 나타낸다). + +## 부하 증가 + +이번에는 부하가 증가함에 따라 오토스케일러가 어떻게 반응하는지를 살펴볼 것이다. 먼저 컨테이너를 하나 실행하고, php-apache 서비스에 무한루프의 쿼리를 전송한다(다른 터미널을 열어 수행하기 바란다). + + +```shell +$ kubectl run -i --tty load-generator --image=busybox /bin/sh + +Hit enter for command prompt + +$ while true; do wget -q -O- http://php-apache.default.svc.cluster.local; done +``` + +실행 후, 약 1분 정도 후에 CPU 부하가 올라가는 것을 볼 수 있다. + +```shell +$ kubectl get hpa +NAME REFERENCE TARGET CURRENT MINPODS MAXPODS REPLICAS AGE +php-apache Deployment/php-apache/scale 305% / 50% 305% 1 10 1 3m + +``` + +CPU 소비가 305%까지 증가하였다. 결과적으로, 디플로이먼트의 레플리카 개수는 7개까지 증가하였다. + +```shell +$ kubectl get deployment php-apache +NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE +php-apache 7 7 7 7 19m +``` + +**참고** 때로는 레플리카의 개수를 안정화시키는데 몇 분이 걸릴 수 있다. 부하의 양은 환경에 따라 다르기 때문에, 최종 레플리카의 개수는 본 예제와 다를 수 있다. + +## 부하 중지 + +본 예제를 마무리하기 위해 부하를 중단시킨다. +`busybox` 컨테이너를 띄운 터미널에서, ` + C`로 부하 발생을 중단시킨다. 그런 다음 (몇 분 후에) 결과를 확인한다. + +```shell +$ kubectl get hpa +NAME REFERENCE TARGET MINPODS MAXPODS REPLICAS AGE +php-apache Deployment/php-apache/scale 0% / 50% 1 10 1 11m + +$ kubectl get deployment php-apache +NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE +php-apache 1 1 1 1 27m +``` + +CPU 사용량은 0으로 떨어졌고, HPA는 레플리카의 개수를 1로 낮췄다. + +{{< note >}} +**참고** 레플리카 오토스케일링은 몇 분 정도 소요된다. +{{< /note >}} + +{{% /capture %}} + +{{% capture discussion %}} + +## 다양한 메트릭 및 사용자 정의 메트릭을 기초로한 오토스케일링 + +`php-apache` 디플로이먼트를 오토스케일링할 때 `autoscaling/v2beta2` API 버전을 사용하여 추가적인 메트릭을 제공할 수 있다. + +첫 번째로, `autoscaling/v2beta2` 형식으로 HorizontalPodAutoscaler YAML 파일을 생성한다. + +```shell +$ kubectl get hpa.v2beta2.autoscaling -o yaml > /tmp/hpa-v2.yaml +``` + +에디터로 `/tmp/hpa-v2.yaml` 파일을 열면, 다음과 같은 YAML이 보일 것입니다. + +```yaml +apiVersion: autoscaling/v2beta2 +kind: HorizontalPodAutoscaler +metadata: + name: php-apache + namespace: default +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: php-apache + minReplicas: 1 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 50 +status: + observedGeneration: 1 + lastScaleTime: + currentReplicas: 1 + desiredReplicas: 1 + currentMetrics: + - type: Resource + resource: + name: cpu + current: + averageUtilization: 0 + averageValue: 0 +``` + +`targetCPUUtilizationPercentage` 필드가 `metrics` 배열로 대체되었다. CPU 사용량 메트릭은 *resource metric* 으로 파드 컨테이너 자원의 백분율로 표현된다. CPU 외에 다른 메트릭을 지정할 수 있는데, 기본적으로 지원되는 다른 메트릭은 메모리뿐이다. 이 자원들은 한 클러스터에서 다른 클러스터로 이름을 변경할 수 없으며, `metrics.k8s.io` API가 가용한 경우 언제든지 사용할 수 있어야 한다. + +또한, `AverageUtilization` 대신 `AverageValue`의 `target` 타입을, 그리고 `target.averageUtilization` 대신 `target.averageValue`로 설정하여 자원 메트릭을 퍼센트 대신 값으로 명시할 수 있다. + +파드 메트릭과 오브젝트 메트릭 두 가지의 *사용자 정의 메트릭* 이 있다. 파드 메트릭과 오브젝트 메트릭. 이 메트릭은 클러스터에 특화된 이름을 가지고 있으며, 더 고급화된 클러스터 모니터링 설정이 필요하다. + +이러한 대체 메트릭 타입중 첫 번째는 *파드 메트릭* 이다. 이 메트릭은 파드들을 설명하고, 파드들간의 평균을 내며, 대상 값과 비교하여 레플리카 개수를 결정한다. + +이것들은 `AverageValue`의 `target`만을 지원한다는 것을 제외하면, 자원 메트릭과 매우 유사하게 동작한다. + +파드 메트릭은 이처럼 메트릭 블록을 사용하여 정의된다. + + +```yaml +type: Pods +pods: + metric: + name: packets-per-second + target: + type: AverageValue + averageValue: 1k +``` + +두 번째 대체 메트릭 타입은 *오브젝트 메트릭*이다. 이 메트릭은 파드를 기술하는 대신에 동일한 네임스페이스 내에 다른 오브젝트를 표현한다. 이 메트릭은 반드시 오브젝트로부터 가져올 필요는 없다. 단지 오브젝트를 기술할 뿐이다. 오브젝트 메트릭은 `Value`과 `AverageValue`의 `target` 타입을 지원한다. `Value`를 사용할 경우 대상은 API로부터 반환되는 메트릭과 직접 비교된다. `AverageValue`를 사용할 경우, 대상 값과 비교되기 이전에 사용자 정의 메트릭 API로부터 반환된 값은 파드의 개수로 나눠진다. 다음은 `requests-per-second` 메트릭을 YAML로 기술한 예제이다. + +```yaml +type: Object +object: + metric: + name: requests-per-second + describedObject: + apiVersion: extensions/v1beta1 + kind: Ingress + name: main-route + target: + type: Value + value: 2k +``` + +이러한 메트릭 블록을 여러 개 제공하면, HorizontalPodAutoscaler는 각 메트릭을 차례로 고려한다. HorizontalPodAutoscaler는 각 메트릭에 대해 제안된 레플리카 개수를 계산하고, 그중 가장 높은 레플리카 개수를 선정한다. + +예를 들어, 네트워크 트래픽 메트릭을 수집하는 모니터링 시스템이 있는 경우, `kubectl edit` 명령어를 이용하여 다음과 같이 정의를 업데이트 할 수 있다. + +```yaml +apiVersion: autoscaling/v2beta1 +kind: HorizontalPodAutoscaler +metadata: + name: php-apache + namespace: default +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: php-apache + minReplicas: 1 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + target: + kind: AverageUtilization + averageUtilization: 50 + - type: Pods + pods: + metric: + name: packets-per-second + targetAverageValue: 1k + - type: Object + object: + metric: + name: requests-per-second + describedObject: + apiVersion: extensions/v1beta1 + kind: Ingress + name: main-route + target: + kind: Value + value: 10k +status: + observedGeneration: 1 + lastScaleTime: + currentReplicas: 1 + desiredReplicas: 1 + currentMetrics: + - type: Resource + resource: + name: cpu + current: + averageUtilization: 0 + averageValue: 0 + - type: Object + object: + metric: + name: requests-per-second + describedObject: + apiVersion: extensions/v1beta1 + kind: Ingress + name: main-route + current: + value: 10k +``` + +이후, HorizontalPodAutoscaler는 각 파드가 요청 된 약 50%의 CPU 사용률을 소모하는지, 초당 1000 패킷을 처리하는지, 메인-루트 인그레스 뒤의 모든 파드들이 초당 10000 요청을 처리하는지 확인한다. + +### 보다 구체적인 메트릭을 기초로한 오토스케일링 + +많은 메트릭 파이프라인들을 사용하면 이름 또는 _labels_ 이라 불리는 추가적인 식별자로 메트릭을 설명할 수 있다. 그리고, 모든 비 자원 메트릭 타입(파드, 오브젝트 그리고 아래 기술된 외부 타입)에 대해, 메트릭 파이프라인으로 전달되는 추가 레이블 셀렉터를 지정할 수 있다. 예를 들면, `verb` 레이블로 `http_requests` 메트릭을 수집하는 경우, 다음과 같이 메트릭 블록을 지정하여 GET 요청에 대해 크기를 조정할 수 있다. + +```yaml +type: Object +object: + metric: + name: `http_requests` + selector: `verb=GET` +``` + +이 셀렉터는 쿠버네티스의 레이블 셀렉터와 동일한 문법이다. 모니터링 파이프라인은 네임과 셀렉터가 여러 시리즈와 일치하는 경우, 해당 여러 시리즈를 단일 값으로 축소하는 방법을 결정한다. 셀렉터는 부가적인 속성이며, 대상 오브젝트(`Pods` 타입의 대상 파드, `Object` 타입으로 기술된 오브젝트)가 아닌 메트릭을 선택할 수 없다. + +### 쿠버네티스 오브젝트와 관련이 없는 메트릭을 기초로한 오토스케일링 + +쿠버네티스 위에서 동작하는 애플리케이션은 쿠버네티스 클러스터의 어떤 오브젝트와도 관련이 없는 메트릭에 기반하여 오토스케일링을 할 수도 있다. 예로, 쿠버네티스 네임스페이스와 관련이 없는 서비스를 기초로한 메트릭을 들 수 있다. 쿠버네티스 버전 1.10 포함 이후 버전에서, *외부 메트릭*을 사용하여 이러한 유스케이스를 해결할 수 있다. + +외부 메트릭 사용시, 먼저 모니터링 시스템에 대한 이해가 있어야 한다. 이 설치는 사용자 정의 메트릭과 유사하다. +외부 메트릭을 사용하면 모니터링 시스템의 사용 가능한 메트릭에 기반하여 클러스터를 오토스케일링 할 수 있다. +위의 예제처럼 `name`과 `selector`를 갖는 `metric` 블록을 제공하고, `Object` 대신에 `External` 메트릭 타입을 사용한다. +만일 여러개의 시계열이 `metricSelector`와 일치하면, HorizontalPodAutoscaler가 값의 합을 사용한다. +외부 메트릭들은 `Value`와 `AverageValue` 대상 타입을 모두 지원하고, `Object` 타입을 사용할 때와 똑같이 동작한다. +예를 들면 애플리케이션이 호스팅 된 대기열 서비스에서 작업을 처리하는 경우, 다음과 같이 HorizontalPodAutoscaler 매니퍼스트에 30개의 미해결 태스크 당 한 개의 워커를 지정하도록 추가할 수 있다. + +```yaml +- type: External + external: + metric: + name: queue_messages_ready + selector: "queue=worker_tasks" + target: + type: AverageValue + averageValue: 30 +``` + +가능하다면, 외부 메트릭 대신 사용자 정의 메트릭 대상 타입을 사용하길 권장한다. 왜냐하면, 클러스터 관리자가 사용자 정의 메트릭 API를 보안관점에서 더 쉽게 보호할 수 있기 때문이다. 외부 메트릭 API는 잠재적으로 어떠한 메트릭에도 접근할 수 있기에, 클러스터 관리자는 API를 노출시킬때 신중해야 한다. + +## 부록: Horizontal Pod Autoscaler 상태 조건 + +HorizontalPodAutoscaler의 `autoscaling/v2beta2` 형식을 사용하면, HorizontalPodAutoscaler에서 쿠버네티스가 설정한 *상태 조건* 을 확인할 수 있다. 이 상태 조건들은 HorizontalPodAutoscaler가 스케일을 할 수 있는지, 어떤 방식으로든 제한되어 있는지 여부를 나타낸다. + +이 조건은 `status.conditions`에 나타난다. HorizontalPodAutoscaler에 영향을 주는 조건을 보기 위해 `kubectl describe hpa`를 사용할 수 있다. + +```shell +$ kubectl describe hpa cm-test +Name: cm-test +Namespace: prom +Labels: +Annotations: +CreationTimestamp: Fri, 16 Jun 2017 18:09:22 +0000 +Reference: ReplicationController/cm-test +Metrics: ( current / target ) + "http_requests" on pods: 66m / 500m +Min replicas: 1 +Max replicas: 4 +ReplicationController pods: 1 current / 1 desired +Conditions: + Type Status Reason Message + ---- ------ ------ ------- + AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale + ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from pods metric http_requests + ScalingLimited False DesiredWithinRange the desired replica count is within the acceptable range +Events: +``` + +이 HorizontalPodAutoscaler 경우, 건강 상태의 여러 조건들을 볼 수 있다. 첫 번째 `AbleToScale`는 HPA가 스케일을 가져오고 업데이트할 수 있는지, 백 오프 관련 조건으로 스케일링이 방지되는지 여부를 나타낸다. 두 번째 `ScalingActive`는 HPA가 활성화되어있는지(즉 대상 레플리카 개수가 0이 아닌지), 원하는 스케일을 계산할 수 있는지 여부를 나타낸다. 만약 `False` 인 경우, 일반적으로 메트릭을 가져오는데 문제가 있다. 마지막으로, 마지막 조건인 `ScalingLimited`는 원하는 스케일 한도가 HorizontalPodAutoscaler의 최대/최소값으로 제한돼있음을 나타낸다. 이는 HorizontalPodAutoscaler에서 레플리카의 개수 제한을 최대/최소값으로 올리거나 낮추려는 것이다. + +## 부록: 수량 + +HorizontalPodAutoscaler와 메트릭 API에서 모든 메트릭은 쿠버네티스에서 사용하는 *수량* 숫자 표기법을 사용한다. 예를 들면, `10500m` 수량은 10진법 `10.5`으로 쓰인다. 메트릭 API들은 가능한 경우 접미사 없이 정수를 반환하며, 일반적으로 수량을 밀리단위로 반환한다. 10진수로 표현했을때, `1`과 `1500m` 또는 `1`과 `1.5` 로 메트릭 값을 나타낼 수 있다. 더 많은 정보를 위해서는 [수량에 관한 용어집](/docs/reference/glossary?core-object=true#term-quantity) 을 참고하기 바란다. + +## 부록: 다른 가능한 시나리오 + +### 명시적으로 오토스케일러 만들기 + +HorizontalPodAutoscaler를 생성하기 위해 `kubectl autoscale` 명령어를 사용하지 않고 명시적으로 다음 파일을 사용하여 만들 수 있다. + +{{< codenew file="application/hpa/php-apache.yaml" >}} + +다음 명령어를 실행하여 오토스케일러를 생성할 것이다. + +```shell +$ kubectl create -f https://k8s.io/examples/application/hpa/php-apache.yaml +horizontalpodautoscaler.autoscaling/php-apache created +``` + +{{% /capture %}} diff --git a/content/ko/docs/tutorials/hello-minikube.md b/content/ko/docs/tutorials/hello-minikube.md index e6cedf8a89..cef4959c23 100644 --- a/content/ko/docs/tutorials/hello-minikube.md +++ b/content/ko/docs/tutorials/hello-minikube.md @@ -37,13 +37,13 @@ menu: {{< note >}} **참고:** macOS 10.13 버전으로 업데이트 후 `brew update`를 실행 시 Homebrew에서 다음과 같은 오류가 발생할 경우에는, - ``` + ```shell Error: /usr/local is not writable. You should change the ownership and permissions of /usr/local back to your user account: sudo chown -R $(whoami) /usr/local ``` Homebrew를 다시 설치하여 문제를 해결할 수 있다. - ``` + ```shell /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" ``` {{< /note >}} @@ -159,15 +159,22 @@ node server.js 8080 포트를 열고, `server.js` 파일을 이미지에 복사하고 Node.js 서버를 시작한다. -이 튜토리얼은 Minikube를 사용하기 때문에, Docker 이미지를 레지스트리로 Push하는 대신, -Minikube VM과 동일한 Docker 호스트를 사용하면 이미지를 단순히 빌드하기만 해도 -이미지가 자동으로 (역주: Minikube에서 사용할 수 있는 위치에) 생긴다. 이를 위해서, -다음의 커맨드를 사용해서 Minikube Docker 데몬을 사용할 수 있도록 한다. +기본적으로, Docker는 로컬 머신의 Docker 레지스트리에 이미지를 생성하고 저장한다. +이 튜토리얼에서는, 로컬 머신의 Docker 레지스트리를 사용하지 않고 Minikube의 +VM 인스턴스 _속에서_ 구동 중인 Docker 데몬의 레지스트리를 사용한다. 'docker' 명령이 +Minikube의 Docker 데몬을 가르키도록 하려면 다음과 같이 입력한다. (unix 셀) ```shell eval $(minikube docker-env) ``` +powershell에서는 다음과 같이 입력한다. +```shell +minikube docker-env | Invoke-Expression +``` + + + {{< note >}} **참고:** 나중에 Minikube 호스트를 더 이상 사용하고 싶지 않은 경우, `eval $ (minikube docker-env -u)`를 실행하여 변경을 되돌릴 수 있다. @@ -179,6 +186,22 @@ Minikube Docker 데몬을 사용하여 Docker 이미지를 빌드한다. (마지 docker build -t hello-node:v1 . ``` +Minikube의 Docker 레지스트리에 이미지가 있는 것을 확인한다. + +```shell +minikube ssh docker images +``` + +Output: + +```shell +REPOSITORY TAG IMAGE ID CREATED SIZE +hello-node v1 f82485ca953c 3 minutes ago 655MB +... +node 6.9.2 faaadb4aaf9b 20 months ago 655MB +``` + + 이제 Minikube VM에서 빌드한 이미지를 실행할 수 있다. ## 디플로이먼트 만들기 diff --git a/content/ko/docs/tutorials/kubernetes-basics/deploy-app/deploy-intro.html b/content/ko/docs/tutorials/kubernetes-basics/deploy-app/deploy-intro.html index 3868b7d180..7b8c344961 100644 --- a/content/ko/docs/tutorials/kubernetes-basics/deploy-app/deploy-intro.html +++ b/content/ko/docs/tutorials/kubernetes-basics/deploy-app/deploy-intro.html @@ -86,11 +86,11 @@ weight: 10

디플로이먼트를 생성할 때, 애플리케이션에 대한 컨테이너 이미지와 구동시키고자 하는 복제 수를 지정해야 한다. 디플로이먼트를 업데이트해서 이런 정보를 나중에 변경할 수 있다. 모듈 - 5와 6의 부트캠프에서 어떻게 + 5와 6의 부트캠프에서 어떻게 스케일하고 업데이트하는지에 대해 다룬다.

+
@@ -103,10 +103,9 @@ weight: 10
-

우리의 첫 번째 디플로이먼트로, Docker 컨테이너로 패키지된 Node.js - 애플리케이션을 사용해보자. 소스 코드와 Dockerfile은 GitHub - 저장소에서 찾을 수 있다.

+

우리의 첫 번째 디플로이먼트로, Docker 컨테이너로 패키지된 Node.js 애플리케이션을 사용해보자. + Node.js 애플리케이션을 작성하고 Docker 컨테이너를 배포하기 위해서, + Hello Minikube 튜토리얼의 지시를 따른다.

이제 디플로이먼트를 이해했으니, 온라인 튜토리얼을 통해 우리의 첫 번째 애플리케이션을 배포해보자!

diff --git a/content/ko/docs/tutorials/kubernetes-basics/update/update-intro.html b/content/ko/docs/tutorials/kubernetes-basics/update/update-intro.html index 0b489f8c43..398fd149e3 100644 --- a/content/ko/docs/tutorials/kubernetes-basics/update/update-intro.html +++ b/content/ko/docs/tutorials/kubernetes-basics/update/update-intro.html @@ -10,6 +10,7 @@ weight: 10 +
diff --git a/content/ko/examples/application/deployment.yaml b/content/ko/examples/application/deployment.yaml new file mode 100644 index 0000000000..0f526b16c0 --- /dev/null +++ b/content/ko/examples/application/deployment.yaml @@ -0,0 +1,19 @@ +apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 +kind: Deployment +metadata: + name: nginx-deployment +spec: + selector: + matchLabels: + app: nginx + replicas: 2 # tells deployment to run 2 pods matching the template + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx:1.7.9 + ports: + - containerPort: 80 diff --git a/content/ko/examples/application/hpa/php-apache.yaml b/content/ko/examples/application/hpa/php-apache.yaml new file mode 100644 index 0000000000..c73ae7d631 --- /dev/null +++ b/content/ko/examples/application/hpa/php-apache.yaml @@ -0,0 +1,13 @@ +apiVersion: autoscaling/v1 +kind: HorizontalPodAutoscaler +metadata: + name: php-apache + namespace: default +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: php-apache + minReplicas: 1 + maxReplicas: 10 + targetCPUUtilizationPercentage: 50 diff --git a/content/ko/examples/minikube/Dockerfile b/content/ko/examples/minikube/Dockerfile index 34b1f40f52..1fe745295a 100644 --- a/content/ko/examples/minikube/Dockerfile +++ b/content/ko/examples/minikube/Dockerfile @@ -1,4 +1,4 @@ -FROM node:6.9.2 +FROM node:6.14.2 EXPOSE 8080 COPY server.js . CMD node server.js diff --git a/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md b/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md new file mode 100644 index 0000000000..32e94830dd --- /dev/null +++ b/content/zh/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md @@ -0,0 +1,203 @@ + + +运行这个命令来搭建Kubernetes master节点 + + +### 简介 + + + +运行这个命令来搭建Kubernetes主节点。 + +``` +kubeadm init [flags] +``` + + +### 参数可选项 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
--apiserver-advertise-address string
API Server将要广播的监听地址。如指定为 `0.0.0.0` 将使用缺省的网卡地址。
--apiserver-bind-port int32     缺省值: 6443
API Server绑定的端口
--apiserver-cert-extra-sans stringSlice
可选的额外提供的证书主题别名（SANs）用于指定API Server的服务器证书。可以是IP地址也可以是DNS名称。
--cert-dir string     缺省值: "/etc/kubernetes/pki"
证书的存储路径。
--config string
kubeadm配置文件的路径。警告：配置文件的功能是实验性的。
--cri-socket string     缺省值: "/var/run/dockershim.sock"
指明要连接的CRI socket文件
--dry-run
不会应用任何改变；只会输出将要执行的操作。
--feature-gates string
键值对的集合，用来控制各种功能的开关。可选项有:
Auditing=true|false (当前为ALPHA状态 - 缺省值=false)
CoreDNS=true|false (缺省值=true)
DynamicKubeletConfig=true|false (当前为BETA状态 - 缺省值=false)
-h, --help
获取init命令的帮助信息
--ignore-preflight-errors stringSlice
忽视检查项错误列表，列表中的每一个检查项如发生错误将被展示输出为警告，而非错误。 例如: 'IsPrivilegedUser,Swap'. 如填写为 'all' 则将忽视所有的检查项错误。
--kubernetes-version string     缺省值: "stable-1"
为control plane选择一个特定的Kubernetes版本。
--node-name string
指定节点的名称。
--pod-network-cidr string
指明pod网络可以使用的IP地址段。 如果设置了这个参数，control plane将会为每一个节点自动分配CIDRs。
--service-cidr string     缺省值: "10.96.0.0/12"
为service的虚拟IP地址另外指定IP地址段
--service-dns-domain string     缺省值: "cluster.local"
为services另外指定域名, 例如： "myorg.internal".
--skip-token-print
不打印出由 `kubeadm init` 命令生成的默认令牌。
--token string
这个令牌用于建立主从节点间的双向受信链接。格式为 [a-z0-9]{6}\.[a-z0-9]{16} - 示例： abcdef.0123456789abcdef
--token-ttl duration     缺省值: 24h0m0s
令牌被自动删除前的可用时长 (示例： 1s, 2m, 3h). 如果设置为 '0', 令牌将永不过期。
+ + + +### 从父命令继承的选项参数 + + + + + + + + + + + + + + + + + +
--rootfs string
[实验性的功能] 相对“真实”宿主机根目录的路径。
+ + + diff --git a/content/zh/docs/reference/setup-tools/kubeadm/kubeadm-init.md b/content/zh/docs/reference/setup-tools/kubeadm/kubeadm-init.md new file mode 100644 index 0000000000..eb4fcea451 --- /dev/null +++ b/content/zh/docs/reference/setup-tools/kubeadm/kubeadm-init.md @@ -0,0 +1,529 @@ +--- +title: kubeadm init +content_template: templates/concept +weight: 20 +--- +{{% capture overview %}} + +此命令初始化一个 Kubernetes master 节点 +{{% /capture %}} + +{{% capture body %}} + +{{< include "generated/kubeadm_init.md" >}} + + +### Init 命令的工作流程 {#init-workflow} + +`kubeadm init` 命令通过执行下列步骤来启动一个 Kubernetes master 节点。 + + +1. 在做出变更前运行一系列的预检项来验证系统状态。 一些检查项目仅仅触发警告，其它的则会被视为错误并且退出 kubeadm，除非问题被解决或者用户指定了 `--ignore-preflight-errors=` 参数。 + + +2. 生成一个自签名的 CA证书 (或者使用现有的证书，如果提供的话) 来为集群中的每一个组件建立身份标识。如果用户已经通过 `--cert-dir` 配置的证书目录（缺省值为 `/etc/kubernetes/pki`）提供了他们自己的 CA证书 以及/或者 密钥， 那么将会跳过这个步骤，正如文档[使用自定义证书](#custom-certificates)中所描述的那样。 + 如果指定了 `--apiserver-cert-extra-sans` 参数, APIServer 的证书将会有额外的 SAN 条目，如果必要的话，将会被转为小写。 + + +1. 将 kubeconfig 文件写入 `/etc/kubernetes/` 目录以便 kubelet、controller-manager 和 scheduler 用来连接到 API server，它们每一个都有自己的身份标识，同时生成一个名为 admin.conf 的独立的 kubeconfig 文件，用于管理操作。 + + +4. 如果 kubeadm 被调用时附带了 `--feature-gates=DynamicKubeletConfig` 参数, + 它会将 kubelet 的初始化配置写入 `/var/lib/kubelet/config/init/kubelet` 文件中。 + 参阅 [通过配置文件设置 Kubelet 参数](/docs/tasks/administer-cluster/kubelet-config-file/)以及 [在一个现有的集群中重新配置节点的 Kubelet 设置](/docs/tasks/administer-cluster/reconfigure-kubelet/)来获取更多关于动态配置 Kubelet 的信息。 + 这个功能现在是默认关闭的，正如你所见它通过一个功能开关控制开闭, 但是在未来的版本中很有可能会默认启用。 + + +5. 为 API server、controller manager 和 scheduler 生成静态 Pod 的清单文件。假使没有提供一个外部的 etcd 服务的话，也会为 etcd 生成一份额外的静态 Pod 清单文件。 + + +静态 Pod 的清单文件被写入到 `/etc/kubernetes/manifests` 目录; kubelet 会监视这个目录以便在系统启动的时候创建 Pods。 + + +一旦 control plane 的 Pods 都运行起来， `kubeadm init` 的工作流程就继续往下执行。 + + +1. 如果 kubeadm 被调用时附带了 `--feature-gates=DynamicKubeletConfig` 参数, + 它将创建一份 ConfigMap 和一些便于 kubelet 访问这份 ConfigMap 的 RBAC 规则，并且通过将 `Node.spec.configSource` 指向到新创建的 ConfigMap 来更新节点设置。这样它就完成了对 Kubelet 的动态配置。 + 这个功能现在是默认关闭的，正如你所见它通过一个功能开关控制开闭, 但是在未来的版本中很有可能会默认启用。 + + +2. 对 master 节点应用 labels 和 taints 以便不会在它上面运行其它的工作负载。 + + +3. 生成令牌以便其它节点以后可以使用这个令牌向 master 节点注册它们自己。 可选的，用户可以通过 `--token` 提供一个令牌, 正如文档[kubeadm 的令牌](/docs/reference/setup-tools/kubeadm/kubeadm-token/) 描述的那样。 + + +4. 为了使得节点能够遵照 [Bootstrap Tokens](/docs/reference/access-authn-authz/bootstrap-tokens/) 和 [TLS Bootstrap](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/)这两份文档中描述的机制加入到集群中，kubeadm 会执行所有的必要配置： + + - 创建一份 ConfigMap 提供添加集群节点所需的信息，并为该 ConfigMap 设置相关的 RBAC 访问规则。 + + - 使得 Bootstrap Tokens 可以访问 CSR 签名 API。 + + - 对新的 CSR 请求配置为自动签发。 + +查阅 [kubeadm join](/docs/reference/setup-tools/kubeadm/kubeadm-join/) 文档以获取更多信息。 + + +1. 通过 API server 安装一个 DNS 服务器 (CoreDNS) 和 kube-proxy 附加组件。 + 在 1.11 版本以及更新版本的 Kubernetes 中 CoreDNS 是默认的 DNS 服务器。 + 如果要安装 kube-dns 而不是 CoreDNS, 你需要在调用 kubeadm 的时候附加 `--feature-gates=CoreDNS=false` 参数。请注意，尽管 DNS 服务器已经被部署了，它并不会被调度直到你安装好了 CNI 网络插件。 + + +2. 如果调用 `kubeadm init` 命令时启用了 alpha 状态的 self-hosting 功能(`--feature-gates=SelfHosting=true`)，基于静态 Pod 的 control plane 将被转换为 [self-hosted control plane](#self-hosting)。 + + +### 结合一份配置文件来使用 kubeadm init {#config-file} + +{{< caution >}} + +**注意:** 配置文件的功能仍然处于 alpha 状态并且在将来的版本中可能会改变。 +{{< /caution >}} + + +通过一份配置文件而不是使用命令行参数来配置 `kubeadm init` 命令是可能的，并且一些更加高级的功能只能够通过配置文件设定。 这份配置文件通过 `--config` 选项参数指定。 + + +在 Kubernetes 1.11 以及之后的版本中，默认的配置可以通过 [kubeadm config print-default](/docs/reference/setup-tools/kubeadm/kubeadm-config/) 命令打印出来。 +**推荐**使用 [kubeadm config migrate](/docs/reference/setup-tools/kubeadm/kubeadm-config/) 命令将你的旧的 `v1alpha3` 版本的配置迁移到 `v1beta1` 版本。因为在 Kubernetes 1.14 版本中将会移除对 `v1alpha3` 这个版本的支持。 + + +如果你想获取 `v1beta1` 版本配置中每个字段的细节说明，你可以查看我们的[API reference 页面]。 (https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1) + + +### 添加 kube-proxy 参数 {#kube-proxy} + + +kubeadm 配置中有关 kube-proxy 的说明请查看: +- [kube-proxy](https://godoc.org/k8s.io/kubernetes/pkg/proxy/apis/config#KubeProxyConfiguration) + + +使用 kubeadm 启用 IPVS 模式的说明请查看: +- [IPVS](https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/README.md) + + +### 向 control plane 组件传递自定义的 flags {#control-plane-flags} + + +有关向 control plane 组件传递 flags 的说明请查看: +- [control-plane-flags](/docs/setup/independent/control-plane-flags/) + + +### 使用自定义的镜像 {#custom-images} + + +默认情况下, kubeadm 会从 `k8s.gcr.io` 仓库拉取镜像, 除非请求的 Kubernetes 版本是一个持续集成版本。这这种情况下，则会使用 `gcr.io/kubernetes-ci-images` 仓库。 + + +你可以通过这份文档里描述的方法 [结合一份配置文件来使用 kubeadm](#config-file) 来改变镜像拉取的策略。 + +允许的自定义功能有: + + +* 提供一个替代的镜像仓库 `imageRepository` 而不是使用 `k8s.gcr.io`。 +* 提供一个统一的 Control Plane 镜像 `unifiedControlPlaneImage` 而不是对每一个 control plane 组件使用不同的镜像。 +* 提供一个指定的 etcd 服务的镜像 `etcd.image` 而不是在 `k8s.gcr.io` 仓库中的可用镜像。 + + +请注意配置文件中的配置项 `kubernetesVersion` 或者命令行参数 `--kubernetes-version` 会影响到镜像的版本。 + + +### 使用自定义的证书 {#custom-certificates} + + +默认情况下, kubeadm 会生成运行一个集群所需的全部证书。 +你可以通过提供你自己的证书来改变这个行为策略。 + + +如果要这样做, 你必须将证书文件放置在通过 `--cert-dir` 命令行参数或者配置文件里的 `CertificatesDir` 配置项指明的目录中。默认的值是 `/etc/kubernetes/pki`。 + + +如果给定的证书和密钥对已经存在，kubeadm 将会跳过生成证书的步骤并且直接将已经存在的文件用于规定的案例中。也就是说你可以拷贝一份已存在的 CA 文件到 `/etc/kubernetes/pki/ca.crt` 和 `/etc/kubernetes/pki/ca.key`，kubeadm将会使用这份 CA 来签发其余的证书。 + + +#### 外部 CA 模式 {#external-ca-mode} + + +如果只提供了 `ca.crt` 文件但是没有提供 `ca.key` 文件也是可以的 (这只对 CA 根证书可用，其它证书不可用)。 +如果所有的其它证书和 kubeconfig 文件已就位， kubeadm 检测到满足以上条件就会激活 "外部 CA" 模式。 kubeadm 将会在没有 CA 密钥文件的情况下继续执行。 + + +否则, kubeadm 将独立运行 controller-manager，附加一个 `--controllers=csrsigner` 的参数，并且指明 CA 证书和密钥。 + + +### 管理 kubeadm 为 kubelet 提供的 systemd 配置文件 {#kubelet-drop-in} + + +kubeadm 包自带了关于 kubelet 应该如何运行的配置文件。请注意 `kubeadm` 客户端命令行工具永远不会修改这份 systemd 配置文件。这份 systemd 配置文件属于 kubeadm deb/rpm 包。 + + +这份文件应该看起来像这样: + + +``` +[Service] +Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf" +Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true" +Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin" +Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local" +Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt" +Environment="KUBELET_CADVISOR_ARGS=" +Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki" +ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS +``` + + +以下是这份文件的详解： + + +* `--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf` 一份 kubeconfig 文件的路径，在加入节点时这份文件被 kubelet 用于获取客户端证书。如果证书获取成功, 一份 kubeconfig 文件将会被写入到由 `--kubeconfig` 参数指明的路径。 + +* `--kubeconfig=/etc/kubernetes/kubelet.conf` 指向 kubeconfig 文件，这份文件为 kubelet 指明 API server 的地址。这份文件也包含了 kubelet 的证书。 + +* `--pod-manifest-path=/etc/kubernetes/manifests` 指明从哪里读取静态 Pod 的清单文件用于启动 control plane。 + +* `--allow-privileged=true` 允许 kubelet 运行 privileged Pods。 + +* `--network-plugin=cni` 使用 CNI 网络。 + +* `--cni-conf-dir=/etc/cni/net.d` 指明从哪里查找[CNI 规格文件](https://github.com/containernetworking/cni/blob/master/SPEC.md)。 + +* `--cni-bin-dir=/opt/cni/bin` 指明从哪里查找实际使用的 CNI 二进制文件。 + +* `--cluster-dns=10.96.0.10` 使用这个集群内部的 DNS 服务器作为 Pods 内 `/etc/resolv.conf` 文件中`nameserver` 的设定值。 + +* `--cluster-domain=cluster.local` 使用这个集群内部的 DNS 域名作为 Pods 内 `/etc/resolv.conf` 文件中`search` 的设定值。 + +* `--client-ca-file=/etc/kubernetes/pki/ca.crt` 使用这个 CA 证书认证发往 Kubelet API 的请求。 + +* `--authorization-mode=Webhook` 通过 `POST` 方法向 API server 发送一个 `SubjectAccessReview` 对象来授权发往 Kubelet API 的请求。 + +* `--rotate-certificates` 当证书临近过期的时候，通过从 `kube-apiserver` 请求新证书来自动替换 kubelet 客户端证书。 + +* `--cert-dir` TLS 证书所在的目录。 + + +### 结合 CRI 运行时使用 kubeadm + + +从v1.6.0版本开始, Kubernetes 默认启用了 CRI, 容器运行时接口。 +默认使用的容器运行时是 Docker, 这通过 `kubelet` 内置的 `dockershim` CRI 实现支持。 + + +其它的基于 CRI 的运行时包括: + +- [cri-containerd](https://github.com/containerd/cri-containerd) +- [cri-o](https://github.com/kubernetes-incubator/cri-o) +- [frakti](https://github.com/kubernetes/frakti) +- [rkt](https://github.com/kubernetes-incubator/rktlet) + + +查阅 [CRI 安装指南](/docs/setup/cri) 获取更多信息。 + + +在你成功安装了 `kubeadm` 和 `kubelet` 工具后, 执行这两个额外的步骤： + + +1. 遵照以上列出的 runtime shim 项目中的安装文档，在每一个节点上安装 runtime shim。 + + +2. 配置 kubelet 使用远程 CRI 运行时。对应你自己环境，请记得改变 `RUNTIME_ENDPOINT` 的值，这个值类似 `/var/run/{your_runtime}.sock`: + +```shell +cat > /etc/systemd/system/kubelet.service.d/20-cri.conf < +现在 `kubelet` 已经准备好使用指定的 CRI 运行时了, 你可以继续使用 `kubeadm init` 和 `kubeadm join` 工作流来部署你的 Kubernetes 集群。 + + +当使用一个外部的 CRI 实现时， 你也许也想为 `kubeadm init` 和 `kubeadm reset` 设置 `--cri-socket`。 + + +### 在你的集群中使用内网 IP + + +为了配置一个 master 与 worker 节点间使用内网 IP 地址通信的集群（而不是使用公网地址），执行以下操作。 + + +1. 当运行 init 命令的时候, 你必须为 API server 指定一个内网 IP 作为监听地址，像这样： + + `kubeadm init --apiserver-advertise-address=` + + +2. 当一个 master 或者 worker 节点可供使用时，向 `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` 文件添加一个标记指明 worker 节点的私有 IP。 + + `--node-ip=` + + +3. 最后, 当你运行 `kubeadm join` 的时候, 确保你提供了正确的 API server 所在的、步骤 1 中定义的私有 IP。 + + +### 设置节点的名称 + + +默认情况下, `kubeadm` 基于机器的 host 地址分配一个节点名称。你可以使用 `--node-name` 参数覆盖这个设置。 + +这个参数会向 kubelet 传递相应的 [`--hostname-override`](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/#options) 参数。 + + +注意覆盖主机名称可能会 [干扰到云服务提供商](https://github.com/kubernetes/website/pull/8873)。 + + +### Self-hosting Kubernetes control plane {#self-hosting} + + +从1.8开始, 你可以试验性地创建一个 _self-hosted_ Kubernetes control plane. 这意味着诸如 API server、controller manager、 以及 scheduler 这些关键组件将作为通过 Kubernetes API 配置的 [DaemonSet pods](/docs/concepts/workloads/controllers/daemonset/) 运行，而不是在 kubelet 中通过静态文件配置的 [static pods](/docs/tasks/administer-cluster/static-pod/)。 + + +若要创建一个 self-hosted 的集群, 向 `kubeadm init` 命令传递 `--feature-gates=SelfHosting=true` 参数。 + +{{< caution >}} + +**小心:** `SelfHosting` 还是一个 alpha 功能。它在 1.12 版本中被标记为废弃，并且将在 1.13 版本中移除。 +{{< /caution >}} + +{{< warning >}} + +**警告:** 查看有关 self-hosted 的注意事项和限制。 +{{< /warning >}} + + +#### 注意事项 + + +1.8 版本中的 Self-hosting 功能有一些重要的限制。特别的, 一个 self-hosted 的集群如果不手动介入的话 _不能够从 master node 的重新启动中恢复_ 。 这个以及其它的一些限制被期望在 self-hosting 功能从 alpha 状态毕业前解决。 + + +默认情况下, self-hosted control plane Pods 依赖位于 [`hostPath`](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) 数据卷中的证书。除了初始化创建证书的过程, 这些证书不被 kubeadm 管理。你可以使用 `--feature-gates=StoreCertsInSecrets=true` 参数来启用一个试验性的模式，在这个模式中 control plane 证书从 Secrets 加载。 这要求对你的集群进行非常小心的对鉴权和授权配置的控制, 并且可能并不适合你的环境。 + +{{< caution >}} + +**小心:** `StoreCertsInSecrets` 是一个 alpha 功能。 它在 1.12 版本中被标记为废弃，并且将在 1.13 版本中移除。 +{{< /caution >}} + + +在 kubeadm 1.8 版本中, control plane 的 self-hosted 的组件并不包含 etcd，它仍然是作为静态 Pod 运行的。 + +#### 流程 + + +self-hosting 集群的启动过程写在了这份 [kubeadm 设计文档](https://github.com/kubernetes/kubeadm/blob/master/docs/design/design_v1.9.md#optional-self-hosting) 文档中。 + + +总的说来, `kubeadm init --feature-gates=SelfHosting=true` 是按照以下步骤工作的: + + + 1. 等待 control plane 的相关组件成功运行起来。这与没有启用 self-hosting 的 `kubeadm init` 指令的流程是一致的。 + + + 2. 使用静态的 control plane Pod 清单文件构建一个 DaemonSet 清单文件用来运行 self-hosted control plane。有时也会在必要的时候修改这些清单文件，比如为 secrets 添加新的数据卷。 + + + 3. 在 `kube-system` 命名空间下创建 DaemonSets 并且等待相应的 Pods 运行起来。 + + + 4. 一旦 self-hosted Pods 可供使用了, 它们相关的静态 Pods 就会被删除并且 kubeadm 继续安装下一个组件。这将触发 kubelet 停止这些静态 Pods。 + + + 5. 当原始的 static control plane 停止的时候, 新创建的 self-hosted control + plane 就能够绑定到端口并且可供使用。 + + +这个过程 （3-6步） 也可以通过 `kubeadm phase selfhosting convert-from-staticpods` 指令触发。 + + +### 在没有互联网连接的情况下运行 kubeadm + + +如果要在没有网络的情况下运行 kubeadm，你需要预先拉取所选版本的主要镜像: + +| Image Name | v1.10 release branch version | +|--------------------------------------------|------------------------------| +| k8s.gcr.io/kube-apiserver-${ARCH} | v1.10.x | +| k8s.gcr.io/kube-controller-manager-${ARCH} | v1.10.x | +| k8s.gcr.io/kube-scheduler-${ARCH} | v1.10.x | +| k8s.gcr.io/kube-proxy-${ARCH} | v1.10.x | +| k8s.gcr.io/etcd-${ARCH} | 3.1.12 | +| k8s.gcr.io/pause-${ARCH} | 3.1 | +| k8s.gcr.io/k8s-dns-sidecar-${ARCH} | 1.14.8 | +| k8s.gcr.io/k8s-dns-kube-dns-${ARCH} | 1.14.8 | +| k8s.gcr.io/k8s-dns-dnsmasq-nanny-${ARCH} | 1.14.8 | +| coredns/coredns | 1.0.6 | + + +此处 `v1.10.x` 意为 "v1.10 分支上的最新的 patch release"。 + + +`${ARCH}` 可以是以下的值: `amd64`, `arm`, `arm64`, `ppc64le` 或者 `s390x`。 + + +如果你运行1.10或者更早版本的 Kubernetes，并且你设置了 `--feature-gates=CoreDNS=true`， +你必须也使用 `coredns/coredns` 镜像, 而不是使用三个 `k8s-dns-*` 镜像。 + + +在 Kubernetes 1.11版本以及之后的版本中，你可以使用 `kubeadm config images` 的子命令来列出和拉取相关镜像: +``` +kubeadm config images list +kubeadm config images pull +``` + + +从 Kubernetes 1.12 版本开始, `k8s.gcr.io/kube-*`、 `k8s.gcr.io/etcd` 和 `k8s.gcr.io/pause` 镜像不再要求 `-${ARCH}` 后缀。 + + +### kubeadm 自动化 + + +与其如文档 [kubeadm 基础教程](/docs/setup/independent/create-cluster-kubeadm/) 所述，将从 `kubeadm init` 取得的令牌拷贝到每一个节点, 倒不如你可以使用更加简单的自动化的方式将令牌并行地分发出去。如果要实现这个自动化，你必须要知道 master node 启动后的 IP 地址。 + + +1. 生成一个令牌. 这个令牌必须具有以下格式：`<6个字符的字符串>.<16字符的字符串>`。更加正式地说法是，它必须符合以下正则表达式： + `[a-z0-9]{6}\.[a-z0-9]{16}`。 + + kubeadm 可以为你生成一个令牌: + + ```bash + kubeadm token generate + ``` + + +2. 使用这个令牌同时启动 master node 和 worker nodes。它们一旦运行起来应该就会互相寻找对方并且建立集群。同样的 `--token` 参数可以同时用于 `kubeadm init` 和 `kubeadm join` 命令。 + + +一旦集群启动起来，你就可以从 master node 的 `/etc/kubernetes/admin.conf` 文件获取管理凭证，然后你就可以使用这个凭证同集群通信了。 + + +注意这种搭建集群的方式在安全保证上会有一些宽松，因为这种方式不允许使用 `--discovery-token-ca-cert-hash` 来验证根 CA 的哈希值（因为当配置节点的时候，它还没有被生成）。如需更多信息，请参阅 [kubeadm join](/docs/reference/setup-tools/kubeadm/kubeadm-join/) 文档。 + +{{% /capture %}} + +{{% capture whatsnext %}} + +* [kubeadm join](/docs/reference/setup-tools/kubeadm/kubeadm-join/) 启动一个 Kubernetes worker node 并且将其加入到集群 +* [kubeadm upgrade](/docs/reference/setup-tools/kubeadm/kubeadm-upgrade/) 将 Kubernetes 集群升级到新版本 +* [kubeadm reset](/docs/reference/setup-tools/kubeadm/kubeadm-reset/) 使用 `kubeadm init` 或者 `kubeadm join`来恢复对节点的改变 +{{% /capture %}} diff --git a/content/zh/docs/reference/setup-tools/kubeadm/kubeadm.md b/content/zh/docs/reference/setup-tools/kubeadm/kubeadm.md new file mode 100644 index 0000000000..5c0b44ab24 --- /dev/null +++ b/content/zh/docs/reference/setup-tools/kubeadm/kubeadm.md @@ -0,0 +1,33 @@ +--- +# title: Overview of kubeadm +title: kubeadm概述 +weight: 10 +--- + + +Kubeadm是一个工具，它提供了 `kubeadm init` 以及 `kubeadm join` 这两个命令作为快速创建kubernetes集群的最佳实践。 + + +kubeadm通过执行必要的操作来启动和运行一个最小可用的集群。它被故意设计为只关心启动集群，而不是之前的节点准备工作。同样的，诸如安装各种各样的可有可无的插件，例如Kubernetes 控制面板、监控解决方案以及特定云提供商的插件，这些都不在它负责的范围。 + + +相反，我们期望由一个基于kubeadm从更高层设计的更加合适的工具来做这些事情；并且，理想情况下，使用kubeadm作为所有部署的基础将会使得创建一个符合期望的集群变得容易。 + +## 接下可以做什么 + + +* [kubeadm init](/docs/reference/setup-tools/kubeadm/kubeadm-init) 启动一个Kubernetes主节点 + +* [kubeadm join](/docs/reference/setup-tools/kubeadm/kubeadm-join) 启动一个Kubernetes工作节点并且将其加入到集群 + +* [kubeadm upgrade](/docs/reference/setup-tools/kubeadm/kubeadm-upgrade) 更新一个Kubernetes集群到新版本 + +* [kubeadm config](/docs/reference/setup-tools/kubeadm/kubeadm-config) 如果你使用kubeadm v1.7.x或者更低版本，你需要对你的集群做一些配置以便使用 `kubeadm upgrade` 命令 + +* [kubeadm token](/docs/reference/setup-tools/kubeadm/kubeadm-token) 使用 `kubeadm join` 来管理令牌 + +* [kubeadm reset](/docs/reference/setup-tools/kubeadm/kubeadm-reset) 使用 `kubeadm init` 或者 `kubeadm join`来恢复对节点的改变 + +* [kubeadm version](/docs/reference/setup-tools/kubeadm/kubeadm-version) 打印出kubeadm版本 + +* [kubeadm alpha](/docs/reference/setup-tools/kubeadm/kubeadm-alpha) 预览一组可用的新功能以便从社区搜集反馈 diff --git a/content/zh/docs/setup/independent/create-cluster-kubeadm.md b/content/zh/docs/setup/independent/create-cluster-kubeadm.md new file mode 100644 index 0000000000..f35508319f --- /dev/null +++ b/content/zh/docs/setup/independent/create-cluster-kubeadm.md @@ -0,0 +1,1115 @@ +--- +title: 使用 kubeadm 创建一个单主集群 +content_template: templates/task +weight: 30 +--- + + + + +{{% capture overview %}} + + +**kubeadm** 能帮助您建立一个小型的符合最佳实践的 Kubernetes 集群。通过使用 kubeadm, 您的集群会符合 [Kubernetes 合规性测试](https://kubernetes.io/blog/2017/10/software-conformance-certification)的要求. Kubeadm 也支持其他的集群生命周期操作，比如升级、降级和管理[启动引导令牌](/docs/reference/access-authn-authz/bootstrap-tokens/)。 + + + +因为您可以在不同类型的机器（比如笔记本、服务器和树莓派等）上安装 kubeadm，因此它非常适合与 Terraform 或 Ansible 这类自动化管理系统集成。 + + + +kubeadm 的简单便捷为大家带来了广泛的用户案例： + +- 新用户可以从 kubeadm 开始来试用 Kubernetes。 +- 熟悉 Kubernetes 的用户可以使用 kubeadm 快速搭建集群并测试他们的应用。 +- 大型的项目可以将 kubeadm 和其他的安装工具一起形成一个比较复杂的系统。 + + + +kubeadm 的设计初衷是为新用户提供一种便捷的方式来首次试用 Kubernetes， +同时也方便老用户搭建集群测试他们的应用。 +此外 kubeadm 也可以跟其它生态系统与/或安装工具集成到一起，提供更强大的功能。 + + + +您可以很方便地在支持 rpm 或 deb 软件包的操作系统上安装 _kubeadm_。对应 kubeadm 的 SIG， +[SIG Cluster Lifecycle](https://github.com/kubernetes/community/tree/master/sig-cluster-lifecycle)， +提供了预编译的这类安装包，当然您也可以自己基于源码为其它操作系统来构造安装包。 + + + +### kubeadm 成熟程度 + +| 功能 | 成熟程度 | +|---------------------------|--------------- | +| 命令行用户体验 | beta | +| 功能实现 | beta | +| 配置文件 API | alpha | +| 自托管 | alpha | +| kubeadm alpha 子命令 | alpha | +| CoreDNS | GA | +| 动态 Kubelet 配置 | alpha | + + + +kubeadm 的整体功能目前还是 **Beta** 状态，然而很快在 2018 年就会转换成**正式发布 (GA)** 状态。 +一些子功能，比如自托管或者配置文件 API 还在开发过程当中。 +随着工具的发展，创建集群的方法可能会有所变化，但是整体部署方案还是比较稳定的。 +在 `kubeadm alpha` 下面的任何命令都只是 alpha 状态，目前只提供初期阶段的服务。 + + + +### 维护周期 + +Kubernetes 发现版本的通常只维护支持九个月，在维护周期内，如果发现有比较重大的 bug 或者安全问题的话， +可能会发布一个补丁版本。下面是 Kubernetes 的发布和维护周期，同时也适用于 `kubeadm`。 + +| Kubernetes 版本 | 发行月份 | 终止维护月份 | +|--------------------|----------------|-------------------| +| v1.6.x | 2017 年 3 月 | 2017 年 12 月 | +| v1.7.x | 2017 年 6 月 | 2018 年 3 月 | +| v1.8.x | 2017 年 9 月 | 2018 年 6 月 | +| v1.9.x | 2017 年 12 月 | 2018 年 9 月 | +| v1.10.x | 2018 年 3 月 | 2018 年 12 月 | +| v1.11.x | 2018 年 6 月 | 2019 年 3 月 | +| v1.12.x | 2018 年 9 月 | 2019 年 6 月 | + +{{% /capture %}} + +{{% capture prerequisites %}} + + +- 一个或者多个兼容 deb 或者 rpm 软件包的操作系统，比如 Ubuntu 或者 CentOS +- 每台机器 2 GB 以上的内存，内存不足时应用会受限制 +- 主节点上 2 CPU 以上 +- 集群里所有的机器有完全的网络连接，公有网络或者私有网络都可以 + +{{% /capture %}} +{{% capture steps %}} + + +## 目标 + +* 搭建一个单主 Kubernetes 集群或者[高可用集群](https://kubernetes.io/docs/setup/independent/high-availability/) +* 在集群上安装 Pod 网络组件以便 Pod 之间可以互相通信 + + + +## 步骤 + +### 在您的机器上安装 kubeadm + +请查阅[安装 kubeadm](/docs/setup/independent/install-kubeadm/)。 + +{{< note >}} +**注意:** 如果您的机器已经安装了 kubeadm, 请运行 `apt-get update && +apt-get upgrade` 或者 `yum update` 来升级至最新版本的 kubeadm. + +升级过程中，kubelet 会每隔几秒钟重启并陷入了不断循环等待 kubeadm 发布指令的状态。 +这个死循环的过程是正常的，当升级并初始化完成您的主节点之后，kubelet 才会正常运行。 +{{< /note >}} + + + +### 初始化您的主节点 + +主节点是集群里运行控制面的机器，包括 etcd (集群的数据库)和 API 服务（kubectl CLI 与之交互）。 + + + + +1. 选择一个 Pod 网络插件，并检查是否在 kubeadm 初始化过程中需要传入什么参数。这个取决于 +您选择的网络插件，您可能需要设置 `--Pod-network-cidr` 来指定网络驱动的 CIDR。请参阅[安装网络插件](#Pod-network)。 +1. (可选) 除非特别指定，kubeadm 会使用默认网关所在的网络接口广播其主节点的 IP 地址。若需使用其他网络接口，请 +给 `kubeadm init` 设置 `--apiserver-advertise-address=` 参数。如果需要部署 +IPv6 的集群，则需要指定一个 IPv6 地址，比如 `--apiserver-advertise-address=fd00::101`。 +1. (可选) 在运行 `kubeadm init` 之前请先执行 `kubeadm config images pull` 来测试与 gcr.io 的连接。 + +现在运行: + +```bash +kubeadm init +``` + + + +### 补充信息 + +想了解更多关于 `kubeadm init` 的参数, 请参阅[kubeadm 参考指南](/docs/reference/setup-tools/kubeadm/kubeadm/)。 + +想了解完整的配置选项，请参阅[配置文件](/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file)。 + +如果想定制控制面组件，包括为活跃性探测和 etcd 服务提供 IPv6 支持以及为各组件提供额外参数，请参阅[定制参数](/docs/admin/kubeadm#custom-args)。 + + + +如果需要再次运行 `kubeadm init`，您必须先[卸载集群](#tear-down)。 + +如果您需要将不同架构的节点加入您的集群，请单独在这类节点上为 `kube-proxy` 和 `kube-dns` 创建 Deployment 或 DaemonSet。 +这是因为这些组件的 Docker 镜像并不支持多架构。 + +`kubeadm init` 首先会执行一系列的运行前检查来确保机器满足运行 Kubernetes 的条件。 +这些检查会抛出警告并在发现错误的时候终止整个初始化进程。 +然后 `kubeadm init` 会下载并安装集群的控制面组件，这可能会花费几分钟时间，其输出如下所示： + +```none +[init] Using Kubernetes version: vX.Y.Z +[preflight] Running pre-flight checks +[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0) +[certificates] Generated ca certificate and key. +[certificates] Generated apiserver certificate and key. +[certificates] apiserver serving cert is signed for DNS names [kubeadm-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.138.0.4] +[certificates] Generated apiserver-kubelet-client certificate and key. +[certificates] Generated sa key and public key. +[certificates] Generated front-proxy-ca certificate and key. +[certificates] Generated front-proxy-client certificate and key. +[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki" +[kubeconfig] Wrote KubeConfig file to disk: "admin.conf" +[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf" +[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf" +[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf" +[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml" +[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml" +[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml" +[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml" +[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests" +[init] This often takes around a minute; or longer if the control plane images have to be pulled. +[apiclient] All control plane components are healthy after 39.511972 seconds +[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace +[markmaster] Will mark node master as master by adding a label and a taint +[markmaster] Master master tainted and labelled with key/value: node-role.kubernetes.io/master="" +[bootstraptoken] Using token: +[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials +[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token +[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace +[addons] Applied essential addon: CoreDNS +[addons] Applied essential addon: kube-proxy + +Your Kubernetes master has initialized successfully! + +To start using your cluster, you need to run (as a regular user): + + mkdir -p $HOME/.kube + sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config + sudo chown $(id -u):$(id -g) $HOME/.kube/config + +You should now deploy a Pod network to the cluster. +Run "kubectl apply -f [Podnetwork].yaml" with one of the addon options listed at: + http://kubernetes.io/docs/admin/addons/ + +You can now join any number of machines by running the following on each node +as root: + + kubeadm join --token : --discovery-token-ca-cert-hash sha256: +``` + + + + +如果需要让普通用户可以运行 kubectl，请运行如下命令，其实这也是 `kubeadm init` 输出的一部分： + +```bash +mkdir -p $HOME/.kube +sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config +sudo chown $(id -u):$(id -g) $HOME/.kube/config +``` + + + +或者，如果您是 `root` 用户，则可以运行： + +```bash +export KUBECONFIG=/etc/kubernetes/admin.conf +``` + + + +请备份好 `kubeadm init` 输出中的 `kubeadm join` 命令，因为您会需要这个命令来[给集群添加节点](#join-nodes)。 + +令牌是主节点和新添加的节点之间进行相互身份验证的，因此请确保其安全。任何人只要知道了这些令牌，就可以随便给您的集群添加节点。 +可以使用 `kubeadm token` 命令来列出、创建和删除这类令牌。 +请参阅[kubeadm 参考指南](/docs/reference/setup-tools/kubeadm/kubeadm-token/)。 + + + +### 安装 Pod 网络插件 {#Pod-network} + +{{< caution >}} +**注意:** 这一节包含了安装和部署顺序的重要信息，执行之前请仔细阅读。 +{{< /caution >}} + +您必须先安装 Pod 网络插件，以便您的 Pod 可以互相通信。 + + + +**网络必须在部署任何应用之前部署好。此外，在网络安装之前是 CoreDNS 不会启用的。 +kubeadm 只支持基于容器网络接口（CNI）的网络而且不支持 kubenet 。** + +有一些项目为 Kubernetes 提供使用 CNI 的 Pod 网络，其中一些也支持[网络策略](/docs/concepts/services-networking/networkpolicies/). +请参阅[插件页面](/docs/concepts/cluster-administration/addons/)了解可用网络插件的完整列表。 +- [CNI v0.6.0](https://github.com/containernetworking/cni/releases/tag/v0.6.0) 也提供了 IPv6 的支持。 +- [CNI 网桥](https://github.com/containernetworking/plugins/blob/master/plugins/main/bridge/README.md) 和 [local-ipam](https://github.com/containernetworking/plugins/blob/master/plugins/ipam/host-local/README.md) 是 Kubernetes 1.9 版本里提供的唯一支持 IPv6 的网络插件。 + + + +注意 kubeadm 默认会创建一个比较安全的集群并强制启用[RBAC](/docs/reference/access-authn-authz/rbac/)。 +请确保您的网络方案支持 RBAC。 + +您可以使用下列命令安装网络插件： + +```bash +kubectl apply -f +``` + + +您仅可以给任何一个集群安装一个网络插件。 + +{{< tabs name="tabs-Pod-install" >}} +{{% tab name="Choose one..." %}} +请选择一个选项来查看对应的第三方网络插件驱动的安装向导。 +{{% /tab %}} + + + +{{% tab name="Calico" %}} +想了解关于 Calico 的使用的更多信息, 请参阅[Kubernetes上的Calico快速实践](https://docs.projectcalico.org/latest/getting-started/kubernetes/)、[安装 Calico 实现网络策略](https://docs.projectcalico.org/latest/getting-started/kubernetes/installation/calico)和其他相关资源。 + +为了 Calico 可以正确工作，您需要给 `kubeadm init` 传递 `--Pod-network-cidr=192.168.0.0/16` 这样的选项， +或者根据您的网络方案更新 `calico.yml` 。注意 Calico 只适用于 `amd64` 架构。 + +```shell +kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml +kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml +``` + +{{% /tab %}} + + + +{{% tab name="Canal" %}} +Canal 使用 Calico 提供的网络策略和 Flannel 提供的网络功能。请查阅 Calico 的官方文档 +[入门指引](https://docs.projectcalico.org/latest/getting-started/kubernetes/installation/flannel)。 + +为了 Canal 可以正确运行，`kubeadm init` 运行时需要设置`--Pod-network-cidr=10.244.0.0/16`，同时注意它只适用于 `amd64` 架构。 + +```shell +kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/rbac.yaml +kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/canal.yaml +``` + +{{% /tab %}} + +{{% tab name="Cilium" %}} +想了解 Kubernetes 上使用 Cilium 的更多相关信息，请查参阅[Kubernetes 上 Cilium 的快速指南](http://docs.cilium.io/en/v1.2/kubernetes/quickinstall/) 和 [Kubernetes 上 Cilium 的安装向导](http://docs.cilium.io/en/v1.2/kubernetes/install/)。 + +尽管这里并不要求给 `kubeadm init` 设置 `--Pod-network-cidr` 参数，但是这是一个高度推荐操作的步骤。 + +这些命令会部署 Cilium 和它自己受 etcd 操作者管理的 etcd。 + +```shell +# 从 Cilium 库下载所需清单文件 +wget https://github.com/cilium/cilium/archive/v1.2.0.zip +unzip v1.2.0.zip +cd cilium-1.2.0/examples/kubernetes/addons/etcd-operator + +# 生成并部署 etcd 证书 +export CLUSTER_DOMAIN=$(kubectl get ConfigMap --namespace kube-system coredns -o yaml | awk '/kubernetes/ {print $2}') +tls/certs/gen-cert.sh $CLUSTER_DOMAIN +tls/deploy-certs.sh + +# 为 kube-dns 设置固定的标识标签 +kubectl label -n kube-system Pod $(kubectl -n kube-system get Pods -l k8s-app=kube-dns -o jsonpath='{range .items[]}{.metadata.name}{" "}{end}') io.cilium.fixed-identity=kube-dns + +kubectl create -f ./ + +# 等待几分钟，Cilium、coredns 和 etcd 的 Pods 会收敛到工作状态 +``` + +{{% /tab %}} + + + +{{% tab name="Flannel" %}} + +为了让 `flannel` 能正确工作，您必须在运行 `kubeadm init` 时设置 `--Pod-network-cidr=10.244.0.0/16`。 + +通过运行 `sysctl net.bridge.bridge-nf-call-iptables=1` 将 `/proc/sys/net/bridge/bridge-nf-call-iptables` 设置成 `1`， +进而确保桥接的 IPv4 流量会传递给 iptables。 +这是一部分 CNI 插件运行的要求条件，请查看[这篇文档](https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/#network-plugin-requirements)获取更详细信息。 + +注意 `flannel` 适用于 `amd64`、`arm`、`arm64` 和 `ppc64le` 架构平台。 + +```shell +kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml +``` + +想了解更多关于 `flannel` 的信息,请查阅[ GitHub 上的 CoreOS flannel 仓库](https://github.com/coreos/flannel)。 +{{% /tab %}} + + + +{{% tab name="Kube-router" %}} +通过运行 `sysctl net.bridge.bridge-nf-call-iptables=1` 将 `/proc/sys/net/bridge/bridge-nf-call-iptables` 设置成 `1`， +确保桥接的 IPv4 流量会传递给 iptables。 +这是一部分 CNI 插件的运行条件。请查看[这篇文档](https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/#network-plugin-requirements)了解更详细的信息。 + +Kube-router 依赖于 kube-controller-manager 来给节点分配 CIDR， 因此需要设置 `kubeadm init` 的 `--Pod-network-cidr` 参数。 + +Kube-router 提供 Pod 间联网、网络策略和和高效的基于 IPVS/LVS 的服务代理功能。 + +想了解关于使用 kubeadm 搭建 Kubernetes 和 Kube-router 的更多信息。请查看官方的[安装指引](https://github.com/cloudnativelabs/kube-router/blob/master/docs/kubeadm.md)。 +{{% /tab %}} + + + +{{% tab name="Romana" %}} +通过运行 `sysctl net.bridge.bridge-nf-call-iptables=1` 将 `/proc/sys/net/bridge/bridge-nf-call-iptables` 设置成 `1`， +确保桥接的 IPv4 流量会传递给 iptables。这是一部分 CNI 插件的运行条件。 +请查看[这篇文档](https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/#network-plugin-requirements) +获取更详细的信息。 + +官方的 Romana 安装指引在[这里](https://github.com/romana/romana/tree/master/containerize#using-kubeadm)。 + +注意，Romana 只适用于 `amd64` 架构。 + +```shell +kubectl apply -f https://raw.githubusercontent.com/romana/romana/master/containerize/specs/romana-kubeadm.yml +``` +{{% /tab %}} + + + +{{% tab name="Weave Net" %}} + +通过运行 `sysctl net.bridge.bridge-nf-call-iptables=1` 将 `/proc/sys/net/bridge/bridge-nf-call-iptables` 设置成 `1`， +将桥接的 IPv4 流量传递给 iptables。这是一部分 CNI 插件的运行条件。 +请查看[这篇文档](https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/#network-plugin-requirements) +获取更详细的信息。 + +官方的 Weave Net 配置向导在[这里](https://www.weave.works/docs/net/latest/kube-addon/)。 + +Weave Net 适用于`amd64`、`arm`、`arm64` 和 `ppc64le` 而不需要其它额外的配置。 +Weave Net 默认启用 hairpin 模式，可以让 Pod 在不知道他们自己的 PodIP 的时候仍可以使用服务的 IP 地址来访问他们自己。 + +```shell +kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')" +``` +{{% /tab %}} + + + +{{% tab name="JuniperContrail/TungstenFabric" %}} +提供了支持 overlay 的 SDN 解决方案，支持多云环境和混合云环境的网络方案，同时支持 overlay 和 underlay、网络策略、 +网络隔离、服务链和灵活的负载均衡。 + +安装 JuniperContrail/TungstenFabric CNI 有很多灵活的方式。 + +请查阅这个[安装指引](https://tungstenfabric.github.io/website/)。 +{{% /tab %}} +{{< /tabs >}} + + + +一旦 Pod 网络安装完成，您就可以通过 `kubectl get Pods --all-namespaces` 的输出来验证 CoreDNS Pod 是否正常运行。 +只要确认了 CoreDNS 正常运行，您就可以向集群中添加节点了。 + +如果您的网络不能工作或者 CoreDNS 不在运行状态，请查阅[查错方案](/docs/setup/independent/troubleshooting-kubeadm/)。 + + + +### 主节点隔离 + +出于安全原因，默认您的主节点不会被调度运行任何 Pod。 +如果您需要在主节点上运行 Pod，比如说部署环境是一个单机器集群，运行： + +```bash +kubectl taint nodes --all node-role.kubernetes.io/master- +``` + +输出类似这样： + +``` +node "test-01" untainted +taint "node-role.kubernetes.io/master:" not found +taint "node-role.kubernetes.io/master:" not found +``` + +这个操作会从任何有 `node-role.kubernetes.io/master` 这种标签的节点移除该标签，包括主节点， +标签的移除意味着集群调度器可以将 Pod 调度到任何节点。 + + + +### 添加节点 {#join-nodes} + +节点就是工作负载（容器和 Pod 等）运行的地方。如需向集群添加新节点，可以在每台机器上面执行如下操作： + +* SSH 连接到机器上 +* 成为 root 用户（比如 `sudo su -`） +* 运行 `kubeadm init` 输出里的命令，即： + +``` bash +kubeadm join --token : --discovery-token-ca-cert-hash sha256: +``` + + + +如果您没有保存令牌的话，可以通过在主节点上执行下面的命令来获取： + +``` bash +kubeadm token list +``` + +输出类似这样: + +``` console +TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS +8ewj1p.9r9hcjoqgajrj4gi 23h 2018-06-12T02:51:28Z authentication, The default bootstrap system: + signing token generated by bootstrappers: + 'kubeadm init'. kubeadm: + default-node-token +``` + + + +默认情况下，令牌会在 24 小时内过期。如果在令牌过期之后添加节点，您可以在主节点上执行下面的命令创建一个新令牌： + +``` bash +kubeadm token create +``` + +输出类似这样： + +``` console +5didvk.d09sbcov8ph2amjw +``` + +如果您也不知道这个 `--discovery-token-ca-cert-hash` 的值，您也可以在主节点上运行下面的命令来获取： + +``` bash +openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \ + openssl dgst -sha256 -hex | sed 's/^.* //' +``` + +输出类似这样： + +``` console +8cb2de97839780a412b93877f8507ad6c94f73add17d5d7058e91741c9d5ec78 +``` + + + +{{< note >}} +**注意:** 若需为 `:` 参数设定一个 IPv6 的元组，地址必须写在一对方括号里面，比如: `[fd00::101]:2073`。 +{{< /note >}} + +输出类似这样: + +``` +[preflight] Running pre-flight checks + +... (log output of join workflow) ... + +Node join complete: +* Certificate signing request sent to master and response + received. +* Kubelet informed of new secure connection details. + +Run 'kubectl get nodes' on the master to see this machine join. +``` + +几秒钟之后，您将能在主节点上的 `kubectl get nodes` 的输出里发现新添加的节点。 + + + +### (可选) 在非主节点上控制集群 + +为了能在其他机器（比如，笔记本）上使用 kubectl 来控制您的集群，您可以从主节点上复制管理员的 +kubeconfig 到您的机器上，像下面这样操作： + +``` bash +scp root@:/etc/kubernetes/admin.conf . +kubectl --kubeconfig ./admin.conf get nodes +``` + +{{< note >}} +**注意:** 上面的例子生效的前提是 SSH 允许 root 用户连接登录。 +如果root 用户不能连接的话，您可以将 `admin.conf` 复制到允许其他用户访问的其他地方并将 `scp` 命令里的用户改成相对应的用户再复制。 + +这个 `admin.conf` 文件给予了用户整个集群的超级用户权限，因此这个操作必须小心谨慎。对于普通用户来说， +更建议创建一个适用于白名单某些权限的验证文件。您可以通过这个命令来生成 `kubeadm alpha phase kubeconfig user --client-name `。 +这个命令会打印 KubeConfig 的内容到标准输出，然后您需要将它保存到一个文件里并分发给您的用户。然后再创建权限的白名单列表， +命令如下： `kubectl create (cluster)rolebinding` 。 +{{< /note >}} + + + +### (可选) 将 API 服务代理到本地 + +如果您需要从集群外部连接到您的 API 服务器，请运行`kubectl proxy`: + +```bash +scp root@:/etc/kubernetes/admin.conf . +kubectl --kubeconfig ./admin.conf proxy +``` + +现在您就可以在本地访问 `http://localhost:8001/api/v1` 来连接 API 服务器了。 + + + +## 卸载集群 {#tear-down} + +想要回退 kubeadm 做出的修改，您需要首先[腾空节点](/docs/reference/generated/kubectl/kubectl-commands#drain) +而且必须确保在关闭节点之前没有任何工作负载在运行。 + +使用正确的登录凭据来连接到主节点： + +```bash +kubectl drain --delete-local-data --force --ignore-daemonsets +kubectl delete node +``` + +然后在待移除的节点上，重置所有 kubeadm 的安装状态： + +```bash +kubeadm reset +``` + +如果您只是想重新运行 `kubeadm init` 或者 `kubeadm join`，[`kubeadm reset`](/docs/reference/setup-tools/kubeadm/kubeadm-reset/)页面有更多的信息可供参考. + + + +## 集群维护 {#lifecycle} + +维护集群（比如升级，降级）的详细指令，可参考[这里](/docs/tasks/administer-cluster/kubeadm)。 + +## 探索其他插件 {#other-addons} + +查看[插件列表](/docs/concepts/cluster-administration/addons/)来发现其他插件，包括日志、监控、网络策略、 +可视化和集群管理工具等等。 + + + +## 后续 {#whats-next} + +* 使用 [Sonobuoy](https://github.com/heptio/sonobuoy) 验证集群是否正确运行 +* 阅读[kubeadm 参考文档](/docs/reference/setup-tools/kubeadm/kubeadm)，学习 kubeadm 的高级应用 +* 进一步了解 Kubernetes 的[概念](/docs/concepts/)和 [`kubectl`](/docs/user-guide/kubectl-overview/) +* 您可以使用 **logrotate** 配置日志轮转。使用 Docker 的时候，您可以给 Docker 守护进程设置日志轮转的选项， +比如 `--log-driver=json-file --log-opt=max-size=10m --log-opt=max-file=5`。请查阅 [Docker 守护进程的配置和纠错](https://docs.docker.com/engine/admin/)。 + +## 反馈 {#feedback} + +* 如果发现故障，请访问 [kubeadm Github issue tracker](https://github.com/kubernetes/kubeadm/issues) +* 如果需要支持，请访问 kubeadm 的 Slack 频道：[#kubeadm](https://kubernetes.slack.com/messages/kubeadm/) +* 访问 SIG cluster-lifecycle 开发者所使用的 Slack 频道： + [#sig-cluster-lifecycle](https://kubernetes.slack.com/messages/sig-cluster-lifecycle/) +* SIG cluster-lifecycle 的 [SIG 信息](#TODO) +* SIG cluster-lifecycle 的邮件列表: + [kubernetes-sig-cluster-lifecycle](https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-lifecycle) + + + +## 版本偏差策略 {#version-skew-policy} + +vX.Y 版本的 kubeadm 命令行工具可能会部署一个控制面版本为 vX.Y 或者 vX.(Y-1) 的集群，也可以用于升级一个 vX.(Y-1) 的由 kubeadm 创建的集群。 +因为我们无法预见未来，版本为 vX.Y 的 kubeadm 可能可以也可能无法用于部署 vX.(Y+1) 版本的集群。 +例子: kubeadm v1.8 可以用于部署 v1.7 和 v1.8 的集群，也可以升级 v1.7 的由 kubeadm 创建的集群到 1.8 版本。 +请查看我们的[安装向导](/docs/setup/independent/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl)，其中提供了关于 kubelet 和控制面版本偏差的更多信息。 + + + +## 跨多平台上使用 kubeadm {#multi-platform} + +kubeadm 的 deb/rpm 包和可执行文件都是适用于 amd64、arm (32位)、arm64、ppc64le 和 s390x等架构平台的， +请查阅[多平台方案](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multi-platform.md)。 + +只有一部分的网络驱动提供了所有平台的网络解决方案，请查询上面的网络驱动列表或者对应的官方文档来确定是否支持您的平台。 + + + +## 局限 {#limitations} + +请注意，kubeadm 还处于正在开发的状态中，这些局限将会在适当的时间修正。 + +1. 这篇文档介绍的创建的集群只能有单一的主节点和单一的 etcd 数据库，这表示如果主节点宕机，您的集群将会丢失数据 + 而且可能无法重新创建。目前给 kubeadm 添加高可用支持（比如多 etcd 多 API服务等等）的功能还在开发当中，因此可先参照下面的 + 临时解决方案: 经常性地[备份 etcd](https://coreos.com/etcd/docs/latest/admin_guide.html)。 + 由 kubeadm 配置的 etcd 数据位于主节点上的 `/var/lib/etcd` 目录。 + +## 查错 {#troubleshooting} + +如果您在使用 kubeadm 发现任何问题，请查阅我们的[纠错文档](/docs/setup/independent/troubleshooting-kubeadm/)。 + + + + + diff --git a/content/zh/docs/setup/independent/install-kubeadm.md b/content/zh/docs/setup/independent/install-kubeadm.md new file mode 100644 index 0000000000..e4016744c4 --- /dev/null +++ b/content/zh/docs/setup/independent/install-kubeadm.md @@ -0,0 +1,397 @@ +--- +title: 安装 kubeadm +content_template: templates/task +weight: 20 +--- + + +{{% capture overview %}} + + + 本文会告诉您如何安装 `kubeadm` 工具。完成本文提到的安装步骤后，您可以阅读 [使用 kubeadm 来创建集群](/docs/setup/independent/create-cluster-kubeadm/) 了解如何使用 kubeadm 来创建集群。 + +{{% /capture %}} + +{{% capture prerequisites %}} + + +* 一台或多台运行着下列系统的机器: + - Ubuntu 16.04+ + - Debian 9 + - CentOS 7 + - RHEL 7 + - Fedora 25/26 (尽力服务) + - HypriotOS v1.0.1+ + - Container Linux (针对1800.6.0 版本测试) +* 每台机器 2 GB 或更多的 RAM (如果少于这个数字将会影响您应用的运行内存) +* 2 CPU 核心或更多 +* 集群中的所有机器的网络彼此均能相互连接(公网和内网都可以) +* 节点之中不可以有重复的主机名，MAC 地址，product_uuid。更多详细信息请参见[这里](#verify-the-mac-address-and-product-uuid-are-unique-for-every-node) 。 +* 开启主机上的一些特定端口. 更多详细信息请参见[这里](#check-required-ports)。 +* 禁用 Swap 交换分区。为了保证 kubelet 正确运行，您 **必须** 禁用交换分区。 + +{{% /capture %}} + +{{% capture steps %}} + + +## 确保每个节点上 MAC 地址和 product_uuid 的唯一性。 + + +* 您可以使用下列命令获取网络接口的 MAC 地址：`ip link` 或是 `ifconfig -a` +* 下列命令可以用来获取 product_uuid `sudo cat /sys/class/dmi/id/product_uuid` + + +一般来讲，硬件设备会拥有独一无二的地址，但是有些虚拟机可能会雷同。Kubernetes 使用这些值来唯一确定集群中的节点。如果这些值在集群中不唯一，可能会导致安装[失败](https://github.com/kubernetes/kubeadm/issues/31)。 + +## 检查网络适配器 + +如果您有一个以上的网络适配器，同时您的 Kubernetes 组件通过默认路由不可达，我们建议您预先添加 IP 路由规则，这样 Kubernetes 集群就可以通过对应的适配器完成连接。 + +## 检查所需端口 + +### Master 节点 +| 规则 | 方向 | 端口范围 | 作用 | 使用者 | +|----------|-----------|------------|-------------------------|---------------------------| +| TCP | Inbound | 6443* | Kubernetes API server | All | +| TCP | Inbound | 2379-2380 | etcd server client API | kube-apiserver, etcd | +| TCP | Inbound | 10250 | Kubelet API | Self, Control plane | +| TCP | Inbound | 10251 | kube-scheduler | Self | +| TCP | Inbound | 10252 | kube-controller-manager | Self | + +### Worker 节点 + +| 规则 | 方向 | 端口范围 | 作用 | 使用者 | +|----------|-----------|-------------|-----------------------|-------------------------| +| TCP | Inbound | 10250 | Kubelet API | Self, Control plane | +| TCP | Inbound | 30000-32767 | NodePort Services** | All | + +** [NodePort 服务](/docs/concepts/services-networking/service/) 的默认端口范围。 + + +任何使用 * 标记的端口号都有可能被覆盖，所以您需要保证您的自定义端口的状态是开放的。 + + +虽然主节点已经包含了 etcd 的端口，您也可以使用自定义的外部 etcd 集群，或是指定自定义端口。 + +您使用的 pod 网络插件 (见下) 也可能需要某些特定端口开启。由于各个 pod 网络插件都有所不同，请参阅他们各自文档中对端口的要求。 + + +## 安装 runtime + + +从 v1.6.0 起，Kubernetes 开始允许使用 CRI，容器运行时接口。默认的容器运行时是 Docker，这是由 `kubelet` 内置的 CRI 实现 `dockershim` 开启的。 + + +其他的容器运行时有： + +- [containerd](https://github.com/containerd/cri) (containerd 的内置 CRI 插件) +- [cri-o](https://github.com/kubernetes-incubator/cri-o) +- [frakti](https://github.com/kubernetes/frakti) +- [rkt](https://github.com/kubernetes-incubator/rktlet) + + +参考 [CRI 安装指南](/docs/setup/cri) 获取更多信息. + + +## 安装 kubeadm, kubelet 和 kubectl + + +您需要在每台机器上都安装以下的软件包： + + * `kubeadm`: 用来初始化集群的指令。 + + * `kubelet`: 在集群中的每个节点上用来启动 pod 和 container 等。 + + * `kubectl`: 用来与集群通信的命令行工具。 + + +kubeadm **不能** 帮您安装或管理 `kubelet` 或 `kubectl` ，所以您得保证他们满足通过 kubeadm 安装的 Kubernetes 控制层对版本的要求。如果版本没有满足要求，就有可能导致一些难以想到的错误或问题。然而控制层与 kubelet 间的 _小版本号_ 不一致无伤大雅，不过请记住 kubelet 的版本不可以超过 API server 的版本。例如 1.8.0 的 API server 可以适配 1.7.0 的 kubelet，反之就不行了。 + + +{{< warning >}} +这些指南不包括所有系统升级时使用的 Kubernetes 程序包。这是因为 kubeadm 和 Kubernetes 需要 [升级时的特别注意事项](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-11/)。 +{{}} + + + +更多关于版本偏差的信息，请参阅 [版本偏差政策](/docs/setup/independent/create-cluster-kubeadm/#version-skew-policy)。 + +{{< tabs name="k8s_install" >}} +{{% tab name="Ubuntu, Debian or HypriotOS" %}} +```bash +apt-get update && apt-get install -y apt-transport-https curl +curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - +cat </etc/apt/sources.list.d/kubernetes.list +deb https://apt.kubernetes.io/ kubernetes-xenial main +EOF +apt-get update +apt-get install -y kubelet kubeadm kubectl +apt-mark hold kubelet kubeadm kubectl +``` +{{% /tab %}} +{{% tab name="CentOS, RHEL or Fedora" %}} +```bash +cat < /etc/yum.repos.d/kubernetes.repo +[kubernetes] +name=Kubernetes +baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 +enabled=1 +gpgcheck=1 +repo_gpgcheck=1 +gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg +exclude=kube* +EOF + + +# 将 SELinux 设置为 permissive 模式(将其禁用) +setenforce 0 +sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config + +yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes + +systemctl enable kubelet && systemctl start kubelet +``` + + + **请注意：** + + + - 通过命令 `setenforce 0` 和 `sed ...` 可以将 SELinux 设置为 permissive 模式(将其禁用)。 + 只有执行这一操作之后，容器才能访问宿主的文件系统，进而能够正常使用 Pod 网络。您必须这么做，直到 kubelet 做出升级支持 SELinux 为止。 + - 一些 RHEL/CentOS 7 的用户曾经遇到过：由于 iptables 被绕过导致网络请求被错误的路由。您得保证 + 在您的 `sysctl` 配置中 `net.bridge.bridge-nf-call-iptables` 被设为1。 + + ```bash + cat < /etc/sysctl.d/k8s.conf + net.bridge.bridge-nf-call-ip6tables = 1 + net.bridge.bridge-nf-call-iptables = 1 + EOF + sysctl --system + ``` +{{% /tab %}} +{{% tab name="Container Linux" %}} + +安装 CNI 插件（大多数 Pod 网络都需要）： + +```bash +CNI_VERSION="v0.6.0" +mkdir -p /opt/cni/bin +curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-amd64-${CNI_VERSION}.tgz" | tar -C /opt/cni/bin -xz +``` + + +安装 crictl (kubeadm / Kubelet 的容器运行时接口 (CRI) 要求) + +```bash +CRICTL_VERSION="v1.11.1" +mkdir -p /opt/bin +curl -L "https://github.com/kubernetes-incubator/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-amd64.tar.gz" | tar -C /opt/bin -xz +``` + + + +安装 `kubeadm`, `kubelet`, `kubectl` 并且添加一个 `kubelet` systemd 服务: + +```bash +RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)" + +mkdir -p /opt/bin +cd /opt/bin +curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/amd64/{kubeadm,kubelet,kubectl} +chmod +x {kubeadm,kubelet,kubectl} + +curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/${RELEASE}/build/debs/kubelet.service" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service +mkdir -p /etc/systemd/system/kubelet.service.d +curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/${RELEASE}/build/debs/10-kubeadm.conf" | sed "s:/usr/bin:/opt/bin:g" > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +``` + + +启用并启动 `kubelet`: + +```bash +systemctl enable kubelet && systemctl start kubelet +``` +{{% /tab %}} +{{< /tabs >}} + + +kubelet 现在每隔几秒就会重启，因为它陷入了一个等待 kubeadm 指令的死循环。 + + +## 在 Master 节点上配置 kubelet 所需的 cgroup 驱动 + + +使用 Docker 时，kubeadm 会自动为其检测 cgroup 驱动在运行时对 `/var/lib/kubelet/kubeadm-flags.env` 文件进行配置。 + +如果您使用了不同的 CRI， 您得把 `/etc/default/kubelet` 文件中的 `cgroup-driver` 位置改为对应的值，像这样： + +```bash +KUBELET_EXTRA_ARGS=--cgroup-driver= +``` + + +这个文件将会被 `kubeadm init` 和 `kubeadm join` 用于为 kubelet 获取 额外的用户参数。 + + +请注意，您**只**需要在您的 cgroup driver 不是 `cgroupfs` 时这么做，因为 `cgroupfs` 已经是 kubelet 的默认值了。 + + +需要重启 kubelet： + +```bash +systemctl daemon-reload +systemctl restart kubelet +``` + + +## 查错 + + +如果您在使用 kubeadm 时候遇到问题，请查看我们的[疑难解答文档](/docs/setup/independent/troubleshooting-kubeadm/). +{{% capture whatsnext %}} + +* [使用 kubeadm 来创建集群](/docs/setup/independent/create-cluster-kubeadm/) + +{{% /capture %}} diff --git a/content/zh/docs/tasks/administer-cluster/encrypt-data.md b/content/zh/docs/tasks/administer-cluster/encrypt-data.md index c454ec9945..d1f12a6edf 100644 --- a/content/zh/docs/tasks/administer-cluster/encrypt-data.md +++ b/content/zh/docs/tasks/administer-cluster/encrypt-data.md @@ -66,8 +66,8 @@ is provided below. ## 理解静态数据加密 ```yaml -kind: EncryptionConfig -apiVersion: v1 +kind: EncryptionConfiguration +apiVersion: apiserver.config.k8s.io/v1 resources: - resources: - secrets @@ -149,8 +149,8 @@ Create a new encryption config file: 创建一个新的加密配置文件： ```yaml -kind: EncryptionConfig -apiVersion: v1 +kind: EncryptionConfiguration +apiVersion: apiserver.config.k8s.io/v1 resources: - resources: - secrets @@ -299,8 +299,8 @@ To disable encryption at rest place the `identity` provider as the first entry i 要禁用 rest 加密，请将 `identity` provider 作为配置中的第一个条目： ```yaml -kind: EncryptionConfig -apiVersion: v1 +kind: EncryptionConfiguration +apiVersion: apiserver.config.k8s.io/v1 resources: - resources: - secrets diff --git a/layouts/shortcodes/deprecationwarning.html b/layouts/shortcodes/deprecationwarning.html index 25bc7e2c41..a3975386e5 100644 --- a/layouts/shortcodes/deprecationwarning.html +++ b/layouts/shortcodes/deprecationwarning.html @@ -4,7 +4,7 @@

Documentation for Kubernetes {{ .Page.Param "version" }} is no longer actively maintained. The version you are currently viewing is a static snapshot. - For up-to-date documentation, see the latest version. + For up-to-date documentation, see the latest version.

diff --git a/static/docs/reference/generated/kubectl/kubectl-commands.html b/static/docs/reference/generated/kubectl/kubectl-commands.html index 22b58af1de..a9dc44c66b 100644 --- a/static/docs/reference/generated/kubectl/kubectl-commands.html +++ b/static/docs/reference/generated/kubectl/kubectl-commands.html @@ -11,7 +11,7 @@ - +
  • example

GETTING STARTED

@@ -84,7 +84,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. raw @@ -210,7 +210,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|templatefile|template|go-template|go-template-file|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. resource @@ -299,13 +299,13 @@ inspect them.

group [] -Groups to bind to the role +Groups to bind to the clusterrole output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. save-config @@ -317,7 +317,7 @@ inspect them.

serviceaccount [] -Service accounts to bind to the role, in the format : +Service accounts to bind to the clusterrole, in the format : template @@ -423,7 +423,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|go-template-file|templatefile|template|go-template|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. save-config @@ -494,7 +494,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|templatefile|template|go-template|go-template-file|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. save-config @@ -575,7 +575,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. save-config @@ -640,7 +640,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. save-config @@ -722,7 +722,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|templatefile|template|go-template|go-template-file|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. save-config @@ -810,7 +810,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|templatefile|template|go-template|go-template-file|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. save-config @@ -892,7 +892,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|go-template-file|templatefile|template|go-template|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. save-config @@ -972,7 +972,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|templatefile|template|go-template|go-template-file|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. resource @@ -1067,7 +1067,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath-file|jsonpath. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. role @@ -1192,7 +1192,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. save-config @@ -1304,7 +1304,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath-file|jsonpath. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. save-config @@ -1394,7 +1394,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. save-config @@ -1475,7 +1475,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. save-config @@ -1553,7 +1553,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. save-config @@ -1624,7 +1624,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. save-config @@ -1701,7 +1701,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|templatefile|template|go-template|go-template-file|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. save-config @@ -1772,7 +1772,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|templatefile|template|go-template|go-template-file|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. save-config @@ -2182,7 +2182,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. overrides @@ -2424,7 +2424,7 @@ inspect them.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. overrides @@ -2635,6 +2635,48 @@ inspect them.

+
+

diff

+
+

Diff resources included in pod.json.

+
+
kubectl diff -f pod.json
+
+
+

Diff file read from stdin

+
+
cat service.yaml | kubectl diff -f -
+
+

Diff configurations specified by filename or stdin between the current online configuration, and the configuration as it would be if applied.

+

Output is always YAML.

+

KUBECTL EXTERNAL DIFF environment variable can be used to select your own diff command. By default, the "diff" command available in your path will be run with "-u" (unicode) and "-N" (treat new files as empty) options.

+

Usage

+

$ diff -f FILENAME

+

Flags

+ + + + + + + + + + + + + + + + + + + + + + + +
NameShorthandDefaultUsage
filenamef[]Filename, directory, or URL to files contains the configuration to diff
recursiveRfalseProcess the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory.

APP MANAGEMENT

This section contains commands for creating, updating, deleting, and viewing your workloads in a Kubernetes cluster.

@@ -2734,7 +2776,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. overwrite @@ -2856,7 +2898,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. record @@ -2943,7 +2985,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|templatefile|template|go-template|go-template-file|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. template @@ -3107,7 +3149,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. overwrite @@ -3160,7 +3202,7 @@ viewing your workloads in a Kubernetes cluster.

kubectl autoscale rc foo --max=5 --cpu-percent=80

Creates an autoscaler that automatically chooses and sets the number of pods that run in a kubernetes cluster.

-

Looks up a Deployment, ReplicaSet, or ReplicationController by name and creates an autoscaler that uses the given resource as a reference. An autoscaler can automatically increase or decrease number of pods deployed within the system as needed.

+

Looks up a Deployment, ReplicaSet, StatefulSet, or ReplicationController by name and creates an autoscaler that uses the given resource as a reference. An autoscaler can automatically increase or decrease number of pods deployed within the system as needed.

Usage

$ autoscale (-f FILENAME | TYPE NAME | TYPE/NAME) [--min=MINPODS] --max=MAXPODS [--cpu-percent=CPU]

Flags

@@ -3226,7 +3268,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. record @@ -3309,7 +3351,7 @@ viewing your workloads in a Kubernetes cluster.

output o yaml -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. output-version @@ -3400,7 +3442,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|templatefile|template|go-template|go-template-file|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. output-patch @@ -3550,7 +3592,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|templatefile|template|go-template|go-template-file|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. overwrite @@ -3661,7 +3703,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. patch @@ -3768,7 +3810,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|go-template|go-template-file|templatefile|template|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. recursive @@ -3871,7 +3913,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. recursive @@ -3931,7 +3973,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath-file|jsonpath. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. recursive @@ -3985,7 +4027,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|go-template|go-template-file|templatefile|template|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. recursive @@ -4108,7 +4150,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. recursive @@ -4201,7 +4243,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. record @@ -4392,7 +4434,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|templatefile|template|go-template|go-template-file|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. overwrite @@ -4510,7 +4552,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|go-template-file|templatefile|template|go-template|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. record @@ -4628,7 +4670,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath-file|jsonpath. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. record @@ -4725,7 +4767,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. record @@ -4821,7 +4863,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. record @@ -4920,7 +4962,7 @@ viewing your workloads in a Kubernetes cluster.

output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. recursive @@ -5006,7 +5048,7 @@ kubectl wait --for output o -Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. recursive @@ -5220,7 +5262,7 @@ applications.

output o -Output format. One of: json|yaml|name|go-template-file|templatefile|template|go-template|jsonpath|jsonpath-file. +Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. recursive @@ -5290,6 +5332,12 @@ applications.

Container name. If omitted, the first container in the pod will be chosen + +no-preserve + +false +The copied file/directory's ownership and permissions will not be preserved in the container +
@@ -5594,6 +5642,16 @@ applications.

kubectl port-forward pod/mypod 8888:5000
+

Listen on port 8888 on all addresses, forwarding to 5000 in the pod

+
+
kubectl port-forward --address 0.0.0.0 pod/mypod 8888:5000
+
+
+

Listen on port 8888 on localhost and selected IP, forwarding to 5000 in the pod

+
+
kubectl port-forward --address localhost,10.19.21.23 pod/mypod 8888:5000
+
+

Listen on a random port locally, forwarding to 5000 in the pod

kubectl port-forward pod/mypod :5000
@@ -5602,7 +5660,7 @@ applications.

 
Use resource type/name such as deployment/mydeployment to select a pod. Resource type defaults to 'pod' if omitted. 

 
If there are multiple pods matching the criteria, a pod will be selected automatically. The forwarding session ends when the selected pod terminates, and rerun of the command is needed to resume forwarding.

 
Usage

-
$ port-forward TYPE/NAME [LOCAL_PORT:]REMOTE_PORT [...[LOCAL_PORT_N:]REMOTE_PORT_N]

+
$ port-forward TYPE/NAME [options] [LOCAL_PORT:]REMOTE_PORT [...[LOCAL_PORT_N:]REMOTE_PORT_N]

 
Flags

 
@@ -5615,6 +5673,12 @@ applications.

+
+
+
+
+
+
@@ -5801,6 +5865,12 @@ applications.

+
+
+
+
+
+
@@ -5883,6 +5953,12 @@ applications.

+
+
+
+
+
+
@@ -5946,7 +6022,7 @@ applications.

-
+
@@ -6001,7 +6077,7 @@ applications.

-
+
@@ -6086,7 +6162,7 @@ applications.

-
+
@@ -6156,7 +6232,7 @@ applications.
$ kubectl drain foo --grace-period=900
 
Drain node in preparation for maintenance. 

-
The given node will be marked unschedulable to prevent new pods from arriving. 'drain' evicts the pods if the APIServer supports eviction (http://kubernetes.io/docs/admin/disruptions/). Otherwise, it will use normal DELETE to delete the pods. The 'drain' evicts or deletes all pods except mirror pods (which cannot be deleted through the API server).  If there are DaemonSet-managed pods, drain will not proceed without --ignore-daemonsets, and regardless it will not delete any DaemonSet-managed pods, because those pods would be immediately replaced by the DaemonSet controller, which ignores unschedulable markings.  If there are any pods that are neither mirror pods nor managed by ReplicationController, ReplicaSet, DaemonSet, StatefulSet or Job, then drain will not delete any pods unless you use --force.  --force will also allow deletion to proceed if the managing resource of one or more pods is missing. 

+
The given node will be marked unschedulable to prevent new pods from arriving. 'drain' evicts the pods if the APIServer supports  http://kubernetes.io/docs/admin/disruptions/. Otherwise, it will use normal DELETE to delete the pods. The 'drain' evicts or deletes all pods except mirror pods (which cannot be deleted through the API server).  If there are DaemonSet-managed pods, drain will not proceed without --ignore-daemonsets, and regardless it will not delete any DaemonSet-managed pods, because those pods would be immediately replaced by the DaemonSet controller, which ignores unschedulable markings.  If there are any pods that are neither mirror pods nor managed by ReplicationController, ReplicaSet, DaemonSet, StatefulSet or Job, then drain will not delete any pods unless you use --force.  --force will also allow deletion to proceed if the managing resource of one or more pods is missing. 
'drain' waits for graceful termination. You should not operate on the machine until the command completes. 
When you are ready to put the node back into service, use kubectl uncordon, which will make the node schedulable again. 
! http://kubernetes.io/images/docs/kubectl_drain.svg

@@ -6283,7 +6359,7 @@ applications.

-
+
@@ -6353,54 +6429,6 @@ applications.
Usage
$ alpha

-
diff

-

-
 Diff resources included in pod.json. By default, it will diff LOCAL and LIVE versions

-

-
kubectl alpha diff -f pod.json
-

-

-
 When one version is specified, diff that version against LIVE

-

-
cat service.yaml | kubectl alpha diff -f - MERGED
-

-

-
 Or specify both versions

-

-
kubectl alpha diff -f pod.json -f service.yaml LAST LOCAL
-

-
Diff configurations specified by filename or stdin between their local, last-applied, live and/or "merged" versions. 

-
LOCAL and LIVE versions are diffed by default. Other available keywords are MERGED and LAST. 

-
Output is always YAML. 

-
KUBERNETES EXTERNAL DIFF environment variable can be used to select your own diff command. By default, the "diff" command available in your path will be run with "-u" (unicode) and "-N" (treat new files as empty) options.

-
Usage

-
$ diff -f FILENAME

-
Flags

-

 
 
 
 
address[localhost]Addresses to listen on (comma separated) 

 pod-running-timeout
 
 1m0s
 Name of Heapster service 
 

 
no-headersfalseIf present, print output without headers 

 selector
 l
 
 Name of Heapster service 
 

 
no-headersfalseIf present, print output without headers. 

 selector
 l
 
 output
 o
 Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. 
 

 

 recursive
 output
 o
 Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. 
 

 

 recursive
 output
 o
 jsonOutput format. One of: json|yaml|name|go-template|go-template-file|templatefile|template|jsonpath-file|jsonpath. Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. 
 

 

 output-directory
 
 
 
 
 
 output
 o
 Output format. One of: json|yaml|name|template|go-template|go-template-file|templatefile|jsonpath|jsonpath-file. Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. 
 

 

 overwrite
 
 
 

-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
NameShorthandDefaultUsage
filenamef[]Filename, directory, or URL to files contains the configuration to diff 
recursiveRfalseProcess the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory. 

-

 
api-resources

 

 
 Print the supported API Resources

@@ -6901,7 +6929,7 @@ source $HOME/.bash_profile
 output
 o
 yaml
-Output format. One of: json|yaml|name|go-template|go-template-file|templatefile|template|jsonpath|jsonpath-file. 
+Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. 
 
 
 raw
@@ -7133,7 +7161,7 @@ source $HOME/.bash_profile
 output
 o
 
-Output format. One of: json|yaml|name|go-template|go-template-file|templatefile|template|jsonpath|jsonpath-file. 
+Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. 
 
 
 poll-interval
@@ -7173,313 +7201,6 @@ source $HOME/.bash_profile
 
 
 
-

-
run-container

-

-
 Start a single instance of nginx.

-

-
kubectl run nginx --image=nginx
-

-

-
 Start a single instance of hazelcast and let the container expose port 5701 .

-

-
kubectl run hazelcast --image=hazelcast --port=5701
-

-

-
 Start a single instance of hazelcast and set environment variables "DNS_DOMAIN=cluster" and "POD_NAMESPACE=default" in the container.

-

-
kubectl run hazelcast --image=hazelcast --env="DNS_DOMAIN=cluster" --env="POD_NAMESPACE=default"
-

-

-
 Start a single instance of hazelcast and set labels "app=hazelcast" and "env=prod" in the container.

-

-
kubectl run hazelcast --image=hazelcast --labels="app=hazelcast,env=prod"
-

-

-
 Start a replicated instance of nginx.

-

-
kubectl run nginx --image=nginx --replicas=5
-

-

-
 Dry run. Print the corresponding API objects without creating them.

-

-
kubectl run nginx --image=nginx --dry-run
-

-

-
 Start a single instance of nginx, but overload the spec of the deployment with a partial set of values parsed from JSON.

-

-
kubectl run nginx --image=nginx --overrides='{ "apiVersion": "v1", "spec": { ... } }'
-

-

-
 Start a pod of busybox and keep it in the foreground, don't restart it if it exits.

-

-
kubectl run -i -t busybox --image=busybox --restart=Never
-

-

-
 Start the nginx container using the default command, but use custom arguments (arg1 .. argN) for that command.

-

-
kubectl run nginx --image=nginx -- <arg1> <arg2> ... <argN>
-

-

-
 Start the nginx container using a different command and custom arguments.

-

-
kubectl run nginx --image=nginx --command -- <cmd> <arg1> ... <argN>
-

-

-
 Start the perl container to compute π to 2000 places and print it out.

-

-
kubectl run pi --image=perl --restart=OnFailure -- perl -Mbignum=bpi -wle 'print bpi(2000)'
-

-

-
 Start the cron job to compute π to 2000 places and print it out every 5 minutes.

-

-
kubectl run pi --schedule="0/5 * * * ?" --image=perl --restart=OnFailure -- perl -Mbignum=bpi -wle 'print bpi(2000)'
-

-
Create and run a particular image, possibly replicated. 

-
Creates a deployment or job to manage the created container(s).

-
Usage

-
$ run-container

-
Flags

-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
NameShorthandDefaultUsage
allow-missing-template-keystrueIf true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats. 
attachfalseIf true, wait for the Pod to start running, and then attach to the Pod as if 'kubectl attach ...' were called.  Default false, unless '-i/--stdin' is set, in which case the default is true. With '--restart=Never' the exit code of the container process is returned. 
cascadetrueIf true, cascade the deletion of the resources managed by this resource (e.g. Pods created by a ReplicationController).  Default true. 
commandfalseIf true and extra arguments are present, use them as the 'command' field in the container, rather than the 'args' field which is the default. 
dry-runfalseIf true, only print the object that would be sent, without sending it. 
env[]Environment variables to set in the container 
exposefalseIf true, a public, external service is created for the container(s) which are run 
filenamef[]to use to replace the resource. 
forcefalseOnly used when grace-period=0. If true, immediately remove resources from API and bypass graceful deletion. Note that immediate deletion of some resources may result in inconsistency or data loss and requires confirmation. 
generatorThe name of the API generator to use, see http://kubernetes.io/docs/user-guide/kubectl-conventions/#generators for a list. 
grace-period-1Period of time in seconds given to the resource to terminate gracefully. Ignored if negative. Set to 1 for immediate shutdown. Can only be set to 0 when --force is true (force deletion). 
hostport-1The host port mapping for the container port. To demonstrate a single-machine container. 
imageThe image for the container to run. 
image-pull-policyThe image pull policy for the container. If left empty, this value will not be specified by the client and defaulted by the server 
labelslComma separated labels to apply to the pod(s). Will override previous values. 
leave-stdin-openfalseIf the pod is started in interactive mode or with stdin, leave stdin open after the first attach completes. By default, stdin will be closed after the first attach completes. 
limitsThe resource requirement limits for this container.  For example, 'cpu=200m,memory=512Mi'.  Note that server side components may assign limits depending on the server configuration, such as limit ranges. 
outputoOutput format. One of: json|yaml|name|templatefile|template|go-template|go-template-file|jsonpath|jsonpath-file. 
overridesAn inline JSON override for the generated object. If this is non-empty, it is used to override the generated object. Requires that the object supply a valid apiVersion field. 
pod-running-timeout1m0sThe length of time (like 5s, 2m, or 3h, higher than zero) to wait until at least one pod is running 
portThe port that this container exposes.  If --expose is true, this is also the port used by the service that is created. 
quietfalseIf true, suppress prompt messages. 
recordfalseRecord current kubectl command in the resource annotation. If set to false, do not record the command. If set to true, record the command. If not set, default to updating the existing annotation value only if one already exists. 
recursiveRfalseProcess the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory. 
replicasr1Number of replicas to create for this container. Default is 1. 
requestsThe resource requirement requests for this container.  For example, 'cpu=100m,memory=256Mi'.  Note that server side components may assign requests depending on the server configuration, such as limit ranges. 
restartAlwaysThe restart policy for this Pod.  Legal values [Always, OnFailure, Never].  If set to 'Always' a deployment is created, if set to 'OnFailure' a job is created, if set to 'Never', a regular pod is created. For the latter two --replicas must be 1.  Default 'Always', for CronJobs Never. 
rmfalseIf true, delete resources created in this command for attached containers. 
save-configfalseIf true, the configuration of current object will be saved in its annotation. Otherwise, the annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future. 
scheduleA schedule in the Cron format the job should be run with. 
service-generatorservice/v2The name of the generator to use for creating a service.  Only used if --expose is true 
service-overridesAn inline JSON override for the generated service object. If this is non-empty, it is used to override the generated object. Requires that the object supply a valid apiVersion field.  Only used if --expose is true. 
serviceaccountService account to set in the pod spec 
stdinifalseKeep stdin open on the container(s) in the pod, even if nothing is attached. 
templateTemplate string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview]. 
timeout0sThe length of time to wait before giving up on a delete, zero means determine a timeout from the size of the object 
ttytfalseAllocated a TTY for each container in the pod. 
waitfalseIf true, wait for resources to be gone before returning. This waits for finalizers.
diff --git a/static/docs/reference/generated/kubectl/navData.js b/static/docs/reference/generated/kubectl/navData.js index 075b2ca9ec..d9ecb0666a 100644 --- a/static/docs/reference/generated/kubectl/navData.js +++ b/static/docs/reference/generated/kubectl/navData.js @@ -1 +1 @@ -(function(){navData = {"toc":[{"section":"run-container","subsections":[]},{"section":"rolling-update","subsections":[]},{"section":"-strong-deprecated-commands-strong-","subsections":[]},{"section":"version","subsections":[]},{"section":"plugin","subsections":[{"section":"-em-list-em-"}]},{"section":"options","subsections":[]},{"section":"explain","subsections":[]},{"section":"config","subsections":[{"section":"-em-view-em-"},{"section":"-em-use-context-em-"},{"section":"-em-unset-em-"},{"section":"-em-set-credentials-em-"},{"section":"-em-set-context-em-"},{"section":"-em-set-cluster-em-"},{"section":"-em-set-em-"},{"section":"-em-rename-context-em-"},{"section":"-em-get-contexts-em-"},{"section":"-em-get-clusters-em-"},{"section":"-em-delete-context-em-"},{"section":"-em-delete-cluster-em-"},{"section":"-em-current-context-em-"}]},{"section":"completion","subsections":[]},{"section":"api-resources","subsections":[]},{"section":"alpha","subsections":[{"section":"-em-diff-em-"}]},{"section":"-strong-kubectl-settings-and-usage-strong-","subsections":[]},{"section":"uncordon","subsections":[]},{"section":"taint","subsections":[]},{"section":"drain","subsections":[]},{"section":"cordon","subsections":[]},{"section":"cluster-info","subsections":[{"section":"-em-dump-em-"}]},{"section":"certificate","subsections":[{"section":"-em-deny-em-"},{"section":"-em-approve-em-"}]},{"section":"api-versions","subsections":[]},{"section":"-strong-cluster-management-strong-","subsections":[]},{"section":"top","subsections":[{"section":"-em-pod-em-"},{"section":"-em-node-em-"}]},{"section":"proxy","subsections":[]},{"section":"port-forward","subsections":[]},{"section":"logs","subsections":[]},{"section":"exec","subsections":[]},{"section":"describe","subsections":[]},{"section":"cp","subsections":[]},{"section":"auth","subsections":[{"section":"-em-reconcile-em-"},{"section":"-em-can-i-em-"}]},{"section":"attach","subsections":[]},{"section":"-strong-working-with-apps-strong-","subsections":[]},{"section":"wait","subsections":[]},{"section":"set","subsections":[{"section":"-em-subject-em-"},{"section":"-em-serviceaccount-em--1"},{"section":"-em-selector-em-"},{"section":"-em-resources-em-"},{"section":"-em-image-em-"},{"section":"-em-env-em-"}]},{"section":"scale","subsections":[]},{"section":"rollout","subsections":[{"section":"-em-undo-em-"},{"section":"-em-status-em-"},{"section":"-em-resume-em-"},{"section":"-em-pause-em-"},{"section":"-em-history-em-"}]},{"section":"replace","subsections":[]},{"section":"patch","subsections":[]},{"section":"label","subsections":[]},{"section":"edit","subsections":[]},{"section":"convert","subsections":[]},{"section":"autoscale","subsections":[]},{"section":"annotate","subsections":[]},{"section":"apply","subsections":[{"section":"-em-view-last-applied-em-"},{"section":"-em-set-last-applied-em-"},{"section":"-em-edit-last-applied-em-"}]},{"section":"-strong-app-management-strong-","subsections":[]},{"section":"delete","subsections":[]},{"section":"expose","subsections":[]},{"section":"run","subsections":[]},{"section":"get","subsections":[]},{"section":"create","subsections":[{"section":"-em-serviceaccount-em-"},{"section":"-em-service-nodeport-em-"},{"section":"-em-service-loadbalancer-em-"},{"section":"-em-service-externalname-em-"},{"section":"-em-service-clusterip-em-"},{"section":"-em-service-em-"},{"section":"-em-secret-tls-em-"},{"section":"-em-secret-generic-em-"},{"section":"-em-secret-docker-registry-em-"},{"section":"-em-secret-em-"},{"section":"-em-rolebinding-em-"},{"section":"-em-role-em-"},{"section":"-em-quota-em-"},{"section":"-em-priorityclass-em-"},{"section":"-em-poddisruptionbudget-em-"},{"section":"-em-namespace-em-"},{"section":"-em-job-em-"},{"section":"-em-deployment-em-"},{"section":"-em-configmap-em-"},{"section":"-em-clusterrolebinding-em-"},{"section":"-em-clusterrole-em-"}]},{"section":"-strong-getting-started-strong-","subsections":[]}],"flatToc":["run-container","rolling-update","-strong-deprecated-commands-strong-","version","-em-list-em-","plugin","options","explain","-em-view-em-","-em-use-context-em-","-em-unset-em-","-em-set-credentials-em-","-em-set-context-em-","-em-set-cluster-em-","-em-set-em-","-em-rename-context-em-","-em-get-contexts-em-","-em-get-clusters-em-","-em-delete-context-em-","-em-delete-cluster-em-","-em-current-context-em-","config","completion","api-resources","-em-diff-em-","alpha","-strong-kubectl-settings-and-usage-strong-","uncordon","taint","drain","cordon","-em-dump-em-","cluster-info","-em-deny-em-","-em-approve-em-","certificate","api-versions","-strong-cluster-management-strong-","-em-pod-em-","-em-node-em-","top","proxy","port-forward","logs","exec","describe","cp","-em-reconcile-em-","-em-can-i-em-","auth","attach","-strong-working-with-apps-strong-","wait","-em-subject-em-","-em-serviceaccount-em--1","-em-selector-em-","-em-resources-em-","-em-image-em-","-em-env-em-","set","scale","-em-undo-em-","-em-status-em-","-em-resume-em-","-em-pause-em-","-em-history-em-","rollout","replace","patch","label","edit","convert","autoscale","annotate","-em-view-last-applied-em-","-em-set-last-applied-em-","-em-edit-last-applied-em-","apply","-strong-app-management-strong-","delete","expose","run","get","-em-serviceaccount-em-","-em-service-nodeport-em-","-em-service-loadbalancer-em-","-em-service-externalname-em-","-em-service-clusterip-em-","-em-service-em-","-em-secret-tls-em-","-em-secret-generic-em-","-em-secret-docker-registry-em-","-em-secret-em-","-em-rolebinding-em-","-em-role-em-","-em-quota-em-","-em-priorityclass-em-","-em-poddisruptionbudget-em-","-em-namespace-em-","-em-job-em-","-em-deployment-em-","-em-configmap-em-","-em-clusterrolebinding-em-","-em-clusterrole-em-","create","-strong-getting-started-strong-"]};})(); \ No newline at end of file +(function(){navData = {"toc":[{"section":"rolling-update","subsections":[]},{"section":"-strong-deprecated-commands-strong-","subsections":[]},{"section":"version","subsections":[]},{"section":"plugin","subsections":[{"section":"-em-list-em-"}]},{"section":"options","subsections":[]},{"section":"explain","subsections":[]},{"section":"config","subsections":[{"section":"-em-view-em-"},{"section":"-em-use-context-em-"},{"section":"-em-unset-em-"},{"section":"-em-set-credentials-em-"},{"section":"-em-set-context-em-"},{"section":"-em-set-cluster-em-"},{"section":"-em-set-em-"},{"section":"-em-rename-context-em-"},{"section":"-em-get-contexts-em-"},{"section":"-em-get-clusters-em-"},{"section":"-em-delete-context-em-"},{"section":"-em-delete-cluster-em-"},{"section":"-em-current-context-em-"}]},{"section":"completion","subsections":[]},{"section":"api-resources","subsections":[]},{"section":"alpha","subsections":[]},{"section":"-strong-kubectl-settings-and-usage-strong-","subsections":[]},{"section":"uncordon","subsections":[]},{"section":"taint","subsections":[]},{"section":"drain","subsections":[]},{"section":"cordon","subsections":[]},{"section":"cluster-info","subsections":[{"section":"-em-dump-em-"}]},{"section":"certificate","subsections":[{"section":"-em-deny-em-"},{"section":"-em-approve-em-"}]},{"section":"api-versions","subsections":[]},{"section":"-strong-cluster-management-strong-","subsections":[]},{"section":"top","subsections":[{"section":"-em-pod-em-"},{"section":"-em-node-em-"}]},{"section":"proxy","subsections":[]},{"section":"port-forward","subsections":[]},{"section":"logs","subsections":[]},{"section":"exec","subsections":[]},{"section":"describe","subsections":[]},{"section":"cp","subsections":[]},{"section":"auth","subsections":[{"section":"-em-reconcile-em-"},{"section":"-em-can-i-em-"}]},{"section":"attach","subsections":[]},{"section":"-strong-working-with-apps-strong-","subsections":[]},{"section":"wait","subsections":[]},{"section":"set","subsections":[{"section":"-em-subject-em-"},{"section":"-em-serviceaccount-em--1"},{"section":"-em-selector-em-"},{"section":"-em-resources-em-"},{"section":"-em-image-em-"},{"section":"-em-env-em-"}]},{"section":"scale","subsections":[]},{"section":"rollout","subsections":[{"section":"-em-undo-em-"},{"section":"-em-status-em-"},{"section":"-em-resume-em-"},{"section":"-em-pause-em-"},{"section":"-em-history-em-"}]},{"section":"replace","subsections":[]},{"section":"patch","subsections":[]},{"section":"label","subsections":[]},{"section":"edit","subsections":[]},{"section":"convert","subsections":[]},{"section":"autoscale","subsections":[]},{"section":"annotate","subsections":[]},{"section":"apply","subsections":[{"section":"-em-view-last-applied-em-"},{"section":"-em-set-last-applied-em-"},{"section":"-em-edit-last-applied-em-"}]},{"section":"-strong-app-management-strong-","subsections":[]},{"section":"diff","subsections":[]},{"section":"delete","subsections":[]},{"section":"expose","subsections":[]},{"section":"run","subsections":[]},{"section":"get","subsections":[]},{"section":"create","subsections":[{"section":"-em-serviceaccount-em-"},{"section":"-em-service-nodeport-em-"},{"section":"-em-service-loadbalancer-em-"},{"section":"-em-service-externalname-em-"},{"section":"-em-service-clusterip-em-"},{"section":"-em-service-em-"},{"section":"-em-secret-tls-em-"},{"section":"-em-secret-generic-em-"},{"section":"-em-secret-docker-registry-em-"},{"section":"-em-secret-em-"},{"section":"-em-rolebinding-em-"},{"section":"-em-role-em-"},{"section":"-em-quota-em-"},{"section":"-em-priorityclass-em-"},{"section":"-em-poddisruptionbudget-em-"},{"section":"-em-namespace-em-"},{"section":"-em-job-em-"},{"section":"-em-deployment-em-"},{"section":"-em-configmap-em-"},{"section":"-em-clusterrolebinding-em-"},{"section":"-em-clusterrole-em-"}]},{"section":"-strong-getting-started-strong-","subsections":[]}],"flatToc":["rolling-update","-strong-deprecated-commands-strong-","version","-em-list-em-","plugin","options","explain","-em-view-em-","-em-use-context-em-","-em-unset-em-","-em-set-credentials-em-","-em-set-context-em-","-em-set-cluster-em-","-em-set-em-","-em-rename-context-em-","-em-get-contexts-em-","-em-get-clusters-em-","-em-delete-context-em-","-em-delete-cluster-em-","-em-current-context-em-","config","completion","api-resources","alpha","-strong-kubectl-settings-and-usage-strong-","uncordon","taint","drain","cordon","-em-dump-em-","cluster-info","-em-deny-em-","-em-approve-em-","certificate","api-versions","-strong-cluster-management-strong-","-em-pod-em-","-em-node-em-","top","proxy","port-forward","logs","exec","describe","cp","-em-reconcile-em-","-em-can-i-em-","auth","attach","-strong-working-with-apps-strong-","wait","-em-subject-em-","-em-serviceaccount-em--1","-em-selector-em-","-em-resources-em-","-em-image-em-","-em-env-em-","set","scale","-em-undo-em-","-em-status-em-","-em-resume-em-","-em-pause-em-","-em-history-em-","rollout","replace","patch","label","edit","convert","autoscale","annotate","-em-view-last-applied-em-","-em-set-last-applied-em-","-em-edit-last-applied-em-","apply","-strong-app-management-strong-","diff","delete","expose","run","get","-em-serviceaccount-em-","-em-service-nodeport-em-","-em-service-loadbalancer-em-","-em-service-externalname-em-","-em-service-clusterip-em-","-em-service-em-","-em-secret-tls-em-","-em-secret-generic-em-","-em-secret-docker-registry-em-","-em-secret-em-","-em-rolebinding-em-","-em-role-em-","-em-quota-em-","-em-priorityclass-em-","-em-poddisruptionbudget-em-","-em-namespace-em-","-em-job-em-","-em-deployment-em-","-em-configmap-em-","-em-clusterrolebinding-em-","-em-clusterrole-em-","create","-strong-getting-started-strong-"]};})(); \ No newline at end of file diff --git a/static/docs/reference/generated/kubectl/node_modules/jquery/dist/jquery.min.js b/static/docs/reference/generated/kubectl/node_modules/jquery/dist/jquery.min.js index 644d35e274..4d9b3a2587 100644 --- a/static/docs/reference/generated/kubectl/node_modules/jquery/dist/jquery.min.js +++ b/static/docs/reference/generated/kubectl/node_modules/jquery/dist/jquery.min.js @@ -1,4 +1,2 @@ -/*! jQuery v3.2.1 | (c) JS Foundation and other contributors | jquery.org/license */ -!function(a,b){"use strict";"object"==typeof module&&"object"==typeof module.exports?module.exports=a.document?b(a,!0):function(a){if(!a.document)throw new Error("jQuery requires a window with a document");return b(a)}:b(a)}("undefined"!=typeof window?window:this,function(a,b){"use strict";var c=[],d=a.document,e=Object.getPrototypeOf,f=c.slice,g=c.concat,h=c.push,i=c.indexOf,j={},k=j.toString,l=j.hasOwnProperty,m=l.toString,n=m.call(Object),o={};function p(a,b){b=b||d;var c=b.createElement("script");c.text=a,b.head.appendChild(c).parentNode.removeChild(c)}var q="3.2.1",r=function(a,b){return new r.fn.init(a,b)},s=/^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g,t=/^-ms-/,u=/-([a-z])/g,v=function(a,b){return b.toUpperCase()};r.fn=r.prototype={jquery:q,constructor:r,length:0,toArray:function(){return f.call(this)},get:function(a){return null==a?f.call(this):a<0?this[a+this.length]:this[a]},pushStack:function(a){var b=r.merge(this.constructor(),a);return b.prevObject=this,b},each:function(a){return r.each(this,a)},map:function(a){return this.pushStack(r.map(this,function(b,c){return a.call(b,c,b)}))},slice:function(){return this.pushStack(f.apply(this,arguments))},first:function(){return this.eq(0)},last:function(){return this.eq(-1)},eq:function(a){var b=this.length,c=+a+(a<0?b:0);return this.pushStack(c>=0&&c0&&b-1 in a)}var x=function(a){var b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u="sizzle"+1*new Date,v=a.document,w=0,x=0,y=ha(),z=ha(),A=ha(),B=function(a,b){return a===b&&(l=!0),0},C={}.hasOwnProperty,D=[],E=D.pop,F=D.push,G=D.push,H=D.slice,I=function(a,b){for(var c=0,d=a.length;c+~]|"+K+")"+K+"*"),S=new RegExp("="+K+"*([^\\]'\"]*?)"+K+"*\\]","g"),T=new RegExp(N),U=new RegExp("^"+L+"$"),V={ID:new RegExp("^#("+L+")"),CLASS:new RegExp("^\\.("+L+")"),TAG:new RegExp("^("+L+"|[*])"),ATTR:new RegExp("^"+M),PSEUDO:new RegExp("^"+N),CHILD:new RegExp("^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\("+K+"*(even|odd|(([+-]|)(\\d*)n|)"+K+"*(?:([+-]|)"+K+"*(\\d+)|))"+K+"*\\)|)","i"),bool:new RegExp("^(?:"+J+")$","i"),needsContext:new RegExp("^"+K+"*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\("+K+"*((?:-\\d)?\\d*)"+K+"*\\)|)(?=[^-]|$)","i")},W=/^(?:input|select|textarea|button)$/i,X=/^h\d$/i,Y=/^[^{]+\{\s*\[native \w/,Z=/^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/,$=/[+~]/,_=new RegExp("\\\\([\\da-f]{1,6}"+K+"?|("+K+")|.)","ig"),aa=function(a,b,c){var d="0x"+b-65536;return d!==d||c?b:d<0?String.fromCharCode(d+65536):String.fromCharCode(d>>10|55296,1023&d|56320)},ba=/([\0-\x1f\x7f]|^-?\d)|^-$|[^\0-\x1f\x7f-\uFFFF\w-]/g,ca=function(a,b){return b?"\0"===a?"\ufffd":a.slice(0,-1)+"\\"+a.charCodeAt(a.length-1).toString(16)+" ":"\\"+a},da=function(){m()},ea=ta(function(a){return a.disabled===!0&&("form"in a||"label"in a)},{dir:"parentNode",next:"legend"});try{G.apply(D=H.call(v.childNodes),v.childNodes),D[v.childNodes.length].nodeType}catch(fa){G={apply:D.length?function(a,b){F.apply(a,H.call(b))}:function(a,b){var c=a.length,d=0;while(a[c++]=b[d++]);a.length=c-1}}}function ga(a,b,d,e){var f,h,j,k,l,o,r,s=b&&b.ownerDocument,w=b?b.nodeType:9;if(d=d||[],"string"!=typeof a||!a||1!==w&&9!==w&&11!==w)return d;if(!e&&((b?b.ownerDocument||b:v)!==n&&m(b),b=b||n,p)){if(11!==w&&(l=Z.exec(a)))if(f=l[1]){if(9===w){if(!(j=b.getElementById(f)))return d;if(j.id===f)return d.push(j),d}else if(s&&(j=s.getElementById(f))&&t(b,j)&&j.id===f)return d.push(j),d}else{if(l[2])return G.apply(d,b.getElementsByTagName(a)),d;if((f=l[3])&&c.getElementsByClassName&&b.getElementsByClassName)return G.apply(d,b.getElementsByClassName(f)),d}if(c.qsa&&!A[a+" "]&&(!q||!q.test(a))){if(1!==w)s=b,r=a;else if("object"!==b.nodeName.toLowerCase()){(k=b.getAttribute("id"))?k=k.replace(ba,ca):b.setAttribute("id",k=u),o=g(a),h=o.length;while(h--)o[h]="#"+k+" "+sa(o[h]);r=o.join(","),s=$.test(a)&&qa(b.parentNode)||b}if(r)try{return G.apply(d,s.querySelectorAll(r)),d}catch(x){}finally{k===u&&b.removeAttribute("id")}}}return i(a.replace(P,"$1"),b,d,e)}function ha(){var a=[];function b(c,e){return a.push(c+" ")>d.cacheLength&&delete b[a.shift()],b[c+" "]=e}return b}function ia(a){return a[u]=!0,a}function ja(a){var b=n.createElement("fieldset");try{return!!a(b)}catch(c){return!1}finally{b.parentNode&&b.parentNode.removeChild(b),b=null}}function ka(a,b){var c=a.split("|"),e=c.length;while(e--)d.attrHandle[c[e]]=b}function la(a,b){var c=b&&a,d=c&&1===a.nodeType&&1===b.nodeType&&a.sourceIndex-b.sourceIndex;if(d)return d;if(c)while(c=c.nextSibling)if(c===b)return-1;return a?1:-1}function ma(a){return function(b){var c=b.nodeName.toLowerCase();return"input"===c&&b.type===a}}function na(a){return function(b){var c=b.nodeName.toLowerCase();return("input"===c||"button"===c)&&b.type===a}}function oa(a){return function(b){return"form"in b?b.parentNode&&b.disabled===!1?"label"in b?"label"in b.parentNode?b.parentNode.disabled===a:b.disabled===a:b.isDisabled===a||b.isDisabled!==!a&&ea(b)===a:b.disabled===a:"label"in b&&b.disabled===a}}function pa(a){return ia(function(b){return b=+b,ia(function(c,d){var e,f=a([],c.length,b),g=f.length;while(g--)c[e=f[g]]&&(c[e]=!(d[e]=c[e]))})})}function qa(a){return a&&"undefined"!=typeof a.getElementsByTagName&&a}c=ga.support={},f=ga.isXML=function(a){var b=a&&(a.ownerDocument||a).documentElement;return!!b&&"HTML"!==b.nodeName},m=ga.setDocument=function(a){var b,e,g=a?a.ownerDocument||a:v;return g!==n&&9===g.nodeType&&g.documentElement?(n=g,o=n.documentElement,p=!f(n),v!==n&&(e=n.defaultView)&&e.top!==e&&(e.addEventListener?e.addEventListener("unload",da,!1):e.attachEvent&&e.attachEvent("onunload",da)),c.attributes=ja(function(a){return a.className="i",!a.getAttribute("className")}),c.getElementsByTagName=ja(function(a){return a.appendChild(n.createComment("")),!a.getElementsByTagName("*").length}),c.getElementsByClassName=Y.test(n.getElementsByClassName),c.getById=ja(function(a){return o.appendChild(a).id=u,!n.getElementsByName||!n.getElementsByName(u).length}),c.getById?(d.filter.ID=function(a){var b=a.replace(_,aa);return function(a){return a.getAttribute("id")===b}},d.find.ID=function(a,b){if("undefined"!=typeof b.getElementById&&p){var c=b.getElementById(a);return c?[c]:[]}}):(d.filter.ID=function(a){var b=a.replace(_,aa);return function(a){var c="undefined"!=typeof a.getAttributeNode&&a.getAttributeNode("id");return c&&c.value===b}},d.find.ID=function(a,b){if("undefined"!=typeof b.getElementById&&p){var c,d,e,f=b.getElementById(a);if(f){if(c=f.getAttributeNode("id"),c&&c.value===a)return[f];e=b.getElementsByName(a),d=0;while(f=e[d++])if(c=f.getAttributeNode("id"),c&&c.value===a)return[f]}return[]}}),d.find.TAG=c.getElementsByTagName?function(a,b){return"undefined"!=typeof b.getElementsByTagName?b.getElementsByTagName(a):c.qsa?b.querySelectorAll(a):void 0}:function(a,b){var c,d=[],e=0,f=b.getElementsByTagName(a);if("*"===a){while(c=f[e++])1===c.nodeType&&d.push(c);return d}return f},d.find.CLASS=c.getElementsByClassName&&function(a,b){if("undefined"!=typeof b.getElementsByClassName&&p)return b.getElementsByClassName(a)},r=[],q=[],(c.qsa=Y.test(n.querySelectorAll))&&(ja(function(a){o.appendChild(a).innerHTML="",a.querySelectorAll("[msallowcapture^='']").length&&q.push("[*^$]="+K+"*(?:''|\"\")"),a.querySelectorAll("[selected]").length||q.push("\\["+K+"*(?:value|"+J+")"),a.querySelectorAll("[id~="+u+"-]").length||q.push("~="),a.querySelectorAll(":checked").length||q.push(":checked"),a.querySelectorAll("a#"+u+"+*").length||q.push(".#.+[+~]")}),ja(function(a){a.innerHTML="";var b=n.createElement("input");b.setAttribute("type","hidden"),a.appendChild(b).setAttribute("name","D"),a.querySelectorAll("[name=d]").length&&q.push("name"+K+"*[*^$|!~]?="),2!==a.querySelectorAll(":enabled").length&&q.push(":enabled",":disabled"),o.appendChild(a).disabled=!0,2!==a.querySelectorAll(":disabled").length&&q.push(":enabled",":disabled"),a.querySelectorAll("*,:x"),q.push(",.*:")})),(c.matchesSelector=Y.test(s=o.matches||o.webkitMatchesSelector||o.mozMatchesSelector||o.oMatchesSelector||o.msMatchesSelector))&&ja(function(a){c.disconnectedMatch=s.call(a,"*"),s.call(a,"[s!='']:x"),r.push("!=",N)}),q=q.length&&new RegExp(q.join("|")),r=r.length&&new RegExp(r.join("|")),b=Y.test(o.compareDocumentPosition),t=b||Y.test(o.contains)?function(a,b){var c=9===a.nodeType?a.documentElement:a,d=b&&b.parentNode;return a===d||!(!d||1!==d.nodeType||!(c.contains?c.contains(d):a.compareDocumentPosition&&16&a.compareDocumentPosition(d)))}:function(a,b){if(b)while(b=b.parentNode)if(b===a)return!0;return!1},B=b?function(a,b){if(a===b)return l=!0,0;var d=!a.compareDocumentPosition-!b.compareDocumentPosition;return d?d:(d=(a.ownerDocument||a)===(b.ownerDocument||b)?a.compareDocumentPosition(b):1,1&d||!c.sortDetached&&b.compareDocumentPosition(a)===d?a===n||a.ownerDocument===v&&t(v,a)?-1:b===n||b.ownerDocument===v&&t(v,b)?1:k?I(k,a)-I(k,b):0:4&d?-1:1)}:function(a,b){if(a===b)return l=!0,0;var c,d=0,e=a.parentNode,f=b.parentNode,g=[a],h=[b];if(!e||!f)return a===n?-1:b===n?1:e?-1:f?1:k?I(k,a)-I(k,b):0;if(e===f)return la(a,b);c=a;while(c=c.parentNode)g.unshift(c);c=b;while(c=c.parentNode)h.unshift(c);while(g[d]===h[d])d++;return d?la(g[d],h[d]):g[d]===v?-1:h[d]===v?1:0},n):n},ga.matches=function(a,b){return ga(a,null,null,b)},ga.matchesSelector=function(a,b){if((a.ownerDocument||a)!==n&&m(a),b=b.replace(S,"='$1']"),c.matchesSelector&&p&&!A[b+" "]&&(!r||!r.test(b))&&(!q||!q.test(b)))try{var d=s.call(a,b);if(d||c.disconnectedMatch||a.document&&11!==a.document.nodeType)return d}catch(e){}return ga(b,n,null,[a]).length>0},ga.contains=function(a,b){return(a.ownerDocument||a)!==n&&m(a),t(a,b)},ga.attr=function(a,b){(a.ownerDocument||a)!==n&&m(a);var e=d.attrHandle[b.toLowerCase()],f=e&&C.call(d.attrHandle,b.toLowerCase())?e(a,b,!p):void 0;return void 0!==f?f:c.attributes||!p?a.getAttribute(b):(f=a.getAttributeNode(b))&&f.specified?f.value:null},ga.escape=function(a){return(a+"").replace(ba,ca)},ga.error=function(a){throw new Error("Syntax error, unrecognized expression: "+a)},ga.uniqueSort=function(a){var b,d=[],e=0,f=0;if(l=!c.detectDuplicates,k=!c.sortStable&&a.slice(0),a.sort(B),l){while(b=a[f++])b===a[f]&&(e=d.push(f));while(e--)a.splice(d[e],1)}return k=null,a},e=ga.getText=function(a){var b,c="",d=0,f=a.nodeType;if(f){if(1===f||9===f||11===f){if("string"==typeof a.textContent)return a.textContent;for(a=a.firstChild;a;a=a.nextSibling)c+=e(a)}else if(3===f||4===f)return a.nodeValue}else while(b=a[d++])c+=e(b);return c},d=ga.selectors={cacheLength:50,createPseudo:ia,match:V,attrHandle:{},find:{},relative:{">":{dir:"parentNode",first:!0}," ":{dir:"parentNode"},"+":{dir:"previousSibling",first:!0},"~":{dir:"previousSibling"}},preFilter:{ATTR:function(a){return a[1]=a[1].replace(_,aa),a[3]=(a[3]||a[4]||a[5]||"").replace(_,aa),"~="===a[2]&&(a[3]=" "+a[3]+" "),a.slice(0,4)},CHILD:function(a){return a[1]=a[1].toLowerCase(),"nth"===a[1].slice(0,3)?(a[3]||ga.error(a[0]),a[4]=+(a[4]?a[5]+(a[6]||1):2*("even"===a[3]||"odd"===a[3])),a[5]=+(a[7]+a[8]||"odd"===a[3])):a[3]&&ga.error(a[0]),a},PSEUDO:function(a){var b,c=!a[6]&&a[2];return V.CHILD.test(a[0])?null:(a[3]?a[2]=a[4]||a[5]||"":c&&T.test(c)&&(b=g(c,!0))&&(b=c.indexOf(")",c.length-b)-c.length)&&(a[0]=a[0].slice(0,b),a[2]=c.slice(0,b)),a.slice(0,3))}},filter:{TAG:function(a){var b=a.replace(_,aa).toLowerCase();return"*"===a?function(){return!0}:function(a){return a.nodeName&&a.nodeName.toLowerCase()===b}},CLASS:function(a){var b=y[a+" "];return b||(b=new RegExp("(^|"+K+")"+a+"("+K+"|$)"))&&y(a,function(a){return b.test("string"==typeof a.className&&a.className||"undefined"!=typeof a.getAttribute&&a.getAttribute("class")||"")})},ATTR:function(a,b,c){return function(d){var e=ga.attr(d,a);return null==e?"!="===b:!b||(e+="","="===b?e===c:"!="===b?e!==c:"^="===b?c&&0===e.indexOf(c):"*="===b?c&&e.indexOf(c)>-1:"$="===b?c&&e.slice(-c.length)===c:"~="===b?(" "+e.replace(O," ")+" ").indexOf(c)>-1:"|="===b&&(e===c||e.slice(0,c.length+1)===c+"-"))}},CHILD:function(a,b,c,d,e){var f="nth"!==a.slice(0,3),g="last"!==a.slice(-4),h="of-type"===b;return 1===d&&0===e?function(a){return!!a.parentNode}:function(b,c,i){var j,k,l,m,n,o,p=f!==g?"nextSibling":"previousSibling",q=b.parentNode,r=h&&b.nodeName.toLowerCase(),s=!i&&!h,t=!1;if(q){if(f){while(p){m=b;while(m=m[p])if(h?m.nodeName.toLowerCase()===r:1===m.nodeType)return!1;o=p="only"===a&&!o&&"nextSibling"}return!0}if(o=[g?q.firstChild:q.lastChild],g&&s){m=q,l=m[u]||(m[u]={}),k=l[m.uniqueID]||(l[m.uniqueID]={}),j=k[a]||[],n=j[0]===w&&j[1],t=n&&j[2],m=n&&q.childNodes[n];while(m=++n&&m&&m[p]||(t=n=0)||o.pop())if(1===m.nodeType&&++t&&m===b){k[a]=[w,n,t];break}}else if(s&&(m=b,l=m[u]||(m[u]={}),k=l[m.uniqueID]||(l[m.uniqueID]={}),j=k[a]||[],n=j[0]===w&&j[1],t=n),t===!1)while(m=++n&&m&&m[p]||(t=n=0)||o.pop())if((h?m.nodeName.toLowerCase()===r:1===m.nodeType)&&++t&&(s&&(l=m[u]||(m[u]={}),k=l[m.uniqueID]||(l[m.uniqueID]={}),k[a]=[w,t]),m===b))break;return t-=e,t===d||t%d===0&&t/d>=0}}},PSEUDO:function(a,b){var c,e=d.pseudos[a]||d.setFilters[a.toLowerCase()]||ga.error("unsupported pseudo: "+a);return e[u]?e(b):e.length>1?(c=[a,a,"",b],d.setFilters.hasOwnProperty(a.toLowerCase())?ia(function(a,c){var d,f=e(a,b),g=f.length;while(g--)d=I(a,f[g]),a[d]=!(c[d]=f[g])}):function(a){return e(a,0,c)}):e}},pseudos:{not:ia(function(a){var b=[],c=[],d=h(a.replace(P,"$1"));return d[u]?ia(function(a,b,c,e){var f,g=d(a,null,e,[]),h=a.length;while(h--)(f=g[h])&&(a[h]=!(b[h]=f))}):function(a,e,f){return b[0]=a,d(b,null,f,c),b[0]=null,!c.pop()}}),has:ia(function(a){return function(b){return ga(a,b).length>0}}),contains:ia(function(a){return a=a.replace(_,aa),function(b){return(b.textContent||b.innerText||e(b)).indexOf(a)>-1}}),lang:ia(function(a){return U.test(a||"")||ga.error("unsupported lang: "+a),a=a.replace(_,aa).toLowerCase(),function(b){var c;do if(c=p?b.lang:b.getAttribute("xml:lang")||b.getAttribute("lang"))return c=c.toLowerCase(),c===a||0===c.indexOf(a+"-");while((b=b.parentNode)&&1===b.nodeType);return!1}}),target:function(b){var c=a.location&&a.location.hash;return c&&c.slice(1)===b.id},root:function(a){return a===o},focus:function(a){return a===n.activeElement&&(!n.hasFocus||n.hasFocus())&&!!(a.type||a.href||~a.tabIndex)},enabled:oa(!1),disabled:oa(!0),checked:function(a){var b=a.nodeName.toLowerCase();return"input"===b&&!!a.checked||"option"===b&&!!a.selected},selected:function(a){return a.parentNode&&a.parentNode.selectedIndex,a.selected===!0},empty:function(a){for(a=a.firstChild;a;a=a.nextSibling)if(a.nodeType<6)return!1;return!0},parent:function(a){return!d.pseudos.empty(a)},header:function(a){return X.test(a.nodeName)},input:function(a){return W.test(a.nodeName)},button:function(a){var b=a.nodeName.toLowerCase();return"input"===b&&"button"===a.type||"button"===b},text:function(a){var b;return"input"===a.nodeName.toLowerCase()&&"text"===a.type&&(null==(b=a.getAttribute("type"))||"text"===b.toLowerCase())},first:pa(function(){return[0]}),last:pa(function(a,b){return[b-1]}),eq:pa(function(a,b,c){return[c<0?c+b:c]}),even:pa(function(a,b){for(var c=0;c=0;)a.push(d);return a}),gt:pa(function(a,b,c){for(var d=c<0?c+b:c;++d1?function(b,c,d){var e=a.length;while(e--)if(!a[e](b,c,d))return!1;return!0}:a[0]}function va(a,b,c){for(var d=0,e=b.length;d-1&&(f[j]=!(g[j]=l))}}else r=wa(r===g?r.splice(o,r.length):r),e?e(null,g,r,i):G.apply(g,r)})}function ya(a){for(var b,c,e,f=a.length,g=d.relative[a[0].type],h=g||d.relative[" "],i=g?1:0,k=ta(function(a){return a===b},h,!0),l=ta(function(a){return I(b,a)>-1},h,!0),m=[function(a,c,d){var e=!g&&(d||c!==j)||((b=c).nodeType?k(a,c,d):l(a,c,d));return b=null,e}];i1&&ua(m),i>1&&sa(a.slice(0,i-1).concat({value:" "===a[i-2].type?"*":""})).replace(P,"$1"),c,i0,e=a.length>0,f=function(f,g,h,i,k){var l,o,q,r=0,s="0",t=f&&[],u=[],v=j,x=f||e&&d.find.TAG("*",k),y=w+=null==v?1:Math.random()||.1,z=x.length;for(k&&(j=g===n||g||k);s!==z&&null!=(l=x[s]);s++){if(e&&l){o=0,g||l.ownerDocument===n||(m(l),h=!p);while(q=a[o++])if(q(l,g||n,h)){i.push(l);break}k&&(w=y)}c&&((l=!q&&l)&&r--,f&&t.push(l))}if(r+=s,c&&s!==r){o=0;while(q=b[o++])q(t,u,g,h);if(f){if(r>0)while(s--)t[s]||u[s]||(u[s]=E.call(i));u=wa(u)}G.apply(i,u),k&&!f&&u.length>0&&r+b.length>1&&ga.uniqueSort(i)}return k&&(w=y,j=v),t};return c?ia(f):f}return h=ga.compile=function(a,b){var c,d=[],e=[],f=A[a+" "];if(!f){b||(b=g(a)),c=b.length;while(c--)f=ya(b[c]),f[u]?d.push(f):e.push(f);f=A(a,za(e,d)),f.selector=a}return f},i=ga.select=function(a,b,c,e){var f,i,j,k,l,m="function"==typeof a&&a,n=!e&&g(a=m.selector||a);if(c=c||[],1===n.length){if(i=n[0]=n[0].slice(0),i.length>2&&"ID"===(j=i[0]).type&&9===b.nodeType&&p&&d.relative[i[1].type]){if(b=(d.find.ID(j.matches[0].replace(_,aa),b)||[])[0],!b)return c;m&&(b=b.parentNode),a=a.slice(i.shift().value.length)}f=V.needsContext.test(a)?0:i.length;while(f--){if(j=i[f],d.relative[k=j.type])break;if((l=d.find[k])&&(e=l(j.matches[0].replace(_,aa),$.test(i[0].type)&&qa(b.parentNode)||b))){if(i.splice(f,1),a=e.length&&sa(i),!a)return G.apply(c,e),c;break}}}return(m||h(a,n))(e,b,!p,c,!b||$.test(a)&&qa(b.parentNode)||b),c},c.sortStable=u.split("").sort(B).join("")===u,c.detectDuplicates=!!l,m(),c.sortDetached=ja(function(a){return 1&a.compareDocumentPosition(n.createElement("fieldset"))}),ja(function(a){return a.innerHTML="","#"===a.firstChild.getAttribute("href")})||ka("type|href|height|width",function(a,b,c){if(!c)return a.getAttribute(b,"type"===b.toLowerCase()?1:2)}),c.attributes&&ja(function(a){return a.innerHTML="",a.firstChild.setAttribute("value",""),""===a.firstChild.getAttribute("value")})||ka("value",function(a,b,c){if(!c&&"input"===a.nodeName.toLowerCase())return a.defaultValue}),ja(function(a){return null==a.getAttribute("disabled")})||ka(J,function(a,b,c){var d;if(!c)return a[b]===!0?b.toLowerCase():(d=a.getAttributeNode(b))&&d.specified?d.value:null}),ga}(a);r.find=x,r.expr=x.selectors,r.expr[":"]=r.expr.pseudos,r.uniqueSort=r.unique=x.uniqueSort,r.text=x.getText,r.isXMLDoc=x.isXML,r.contains=x.contains,r.escapeSelector=x.escape;var y=function(a,b,c){var d=[],e=void 0!==c;while((a=a[b])&&9!==a.nodeType)if(1===a.nodeType){if(e&&r(a).is(c))break;d.push(a)}return d},z=function(a,b){for(var c=[];a;a=a.nextSibling)1===a.nodeType&&a!==b&&c.push(a);return c},A=r.expr.match.needsContext;function B(a,b){return a.nodeName&&a.nodeName.toLowerCase()===b.toLowerCase()}var C=/^<([a-z][^\/\0>:\x20\t\r\n\f]*)[\x20\t\r\n\f]*\/?>(?:<\/\1>|)$/i,D=/^.[^:#\[\.,]*$/;function E(a,b,c){return r.isFunction(b)?r.grep(a,function(a,d){return!!b.call(a,d,a)!==c}):b.nodeType?r.grep(a,function(a){return a===b!==c}):"string"!=typeof b?r.grep(a,function(a){return i.call(b,a)>-1!==c}):D.test(b)?r.filter(b,a,c):(b=r.filter(b,a),r.grep(a,function(a){return i.call(b,a)>-1!==c&&1===a.nodeType}))}r.filter=function(a,b,c){var d=b[0];return c&&(a=":not("+a+")"),1===b.length&&1===d.nodeType?r.find.matchesSelector(d,a)?[d]:[]:r.find.matches(a,r.grep(b,function(a){return 1===a.nodeType}))},r.fn.extend({find:function(a){var b,c,d=this.length,e=this;if("string"!=typeof a)return this.pushStack(r(a).filter(function(){for(b=0;b1?r.uniqueSort(c):c},filter:function(a){return this.pushStack(E(this,a||[],!1))},not:function(a){return this.pushStack(E(this,a||[],!0))},is:function(a){return!!E(this,"string"==typeof a&&A.test(a)?r(a):a||[],!1).length}});var F,G=/^(?:\s*(<[\w\W]+>)[^>]*|#([\w-]+))$/,H=r.fn.init=function(a,b,c){var e,f;if(!a)return this;if(c=c||F,"string"==typeof a){if(e="<"===a[0]&&">"===a[a.length-1]&&a.length>=3?[null,a,null]:G.exec(a),!e||!e[1]&&b)return!b||b.jquery?(b||c).find(a):this.constructor(b).find(a);if(e[1]){if(b=b instanceof r?b[0]:b,r.merge(this,r.parseHTML(e[1],b&&b.nodeType?b.ownerDocument||b:d,!0)),C.test(e[1])&&r.isPlainObject(b))for(e in b)r.isFunction(this[e])?this[e](b[e]):this.attr(e,b[e]);return this}return f=d.getElementById(e[2]),f&&(this[0]=f,this.length=1),this}return a.nodeType?(this[0]=a,this.length=1,this):r.isFunction(a)?void 0!==c.ready?c.ready(a):a(r):r.makeArray(a,this)};H.prototype=r.fn,F=r(d);var I=/^(?:parents|prev(?:Until|All))/,J={children:!0,contents:!0,next:!0,prev:!0};r.fn.extend({has:function(a){var b=r(a,this),c=b.length;return this.filter(function(){for(var a=0;a-1:1===c.nodeType&&r.find.matchesSelector(c,a))){f.push(c);break}return this.pushStack(f.length>1?r.uniqueSort(f):f)},index:function(a){return a?"string"==typeof a?i.call(r(a),this[0]):i.call(this,a.jquery?a[0]:a):this[0]&&this[0].parentNode?this.first().prevAll().length:-1},add:function(a,b){return this.pushStack(r.uniqueSort(r.merge(this.get(),r(a,b))))},addBack:function(a){return this.add(null==a?this.prevObject:this.prevObject.filter(a))}});function K(a,b){while((a=a[b])&&1!==a.nodeType);return a}r.each({parent:function(a){var b=a.parentNode;return b&&11!==b.nodeType?b:null},parents:function(a){return y(a,"parentNode")},parentsUntil:function(a,b,c){return y(a,"parentNode",c)},next:function(a){return K(a,"nextSibling")},prev:function(a){return K(a,"previousSibling")},nextAll:function(a){return y(a,"nextSibling")},prevAll:function(a){return y(a,"previousSibling")},nextUntil:function(a,b,c){return y(a,"nextSibling",c)},prevUntil:function(a,b,c){return y(a,"previousSibling",c)},siblings:function(a){return z((a.parentNode||{}).firstChild,a)},children:function(a){return z(a.firstChild)},contents:function(a){return B(a,"iframe")?a.contentDocument:(B(a,"template")&&(a=a.content||a),r.merge([],a.childNodes))}},function(a,b){r.fn[a]=function(c,d){var e=r.map(this,b,c);return"Until"!==a.slice(-5)&&(d=c),d&&"string"==typeof d&&(e=r.filter(d,e)),this.length>1&&(J[a]||r.uniqueSort(e),I.test(a)&&e.reverse()),this.pushStack(e)}});var L=/[^\x20\t\r\n\f]+/g;function M(a){var b={};return r.each(a.match(L)||[],function(a,c){b[c]=!0}),b}r.Callbacks=function(a){a="string"==typeof a?M(a):r.extend({},a);var b,c,d,e,f=[],g=[],h=-1,i=function(){for(e=e||a.once,d=b=!0;g.length;h=-1){c=g.shift();while(++h-1)f.splice(c,1),c<=h&&h--}),this},has:function(a){return a?r.inArray(a,f)>-1:f.length>0},empty:function(){return f&&(f=[]),this},disable:function(){return e=g=[],f=c="",this},disabled:function(){return!f},lock:function(){return e=g=[],c||b||(f=c=""),this},locked:function(){return!!e},fireWith:function(a,c){return e||(c=c||[],c=[a,c.slice?c.slice():c],g.push(c),b||i()),this},fire:function(){return j.fireWith(this,arguments),this},fired:function(){return!!d}};return j};function N(a){return a}function O(a){throw a}function P(a,b,c,d){var e;try{a&&r.isFunction(e=a.promise)?e.call(a).done(b).fail(c):a&&r.isFunction(e=a.then)?e.call(a,b,c):b.apply(void 0,[a].slice(d))}catch(a){c.apply(void 0,[a])}}r.extend({Deferred:function(b){var c=[["notify","progress",r.Callbacks("memory"),r.Callbacks("memory"),2],["resolve","done",r.Callbacks("once memory"),r.Callbacks("once memory"),0,"resolved"],["reject","fail",r.Callbacks("once memory"),r.Callbacks("once memory"),1,"rejected"]],d="pending",e={state:function(){return d},always:function(){return f.done(arguments).fail(arguments),this},"catch":function(a){return e.then(null,a)},pipe:function(){var a=arguments;return r.Deferred(function(b){r.each(c,function(c,d){var e=r.isFunction(a[d[4]])&&a[d[4]];f[d[1]](function(){var a=e&&e.apply(this,arguments);a&&r.isFunction(a.promise)?a.promise().progress(b.notify).done(b.resolve).fail(b.reject):b[d[0]+"With"](this,e?[a]:arguments)})}),a=null}).promise()},then:function(b,d,e){var f=0;function g(b,c,d,e){return function(){var h=this,i=arguments,j=function(){var a,j;if(!(b=f&&(d!==O&&(h=void 0,i=[a]),c.rejectWith(h,i))}};b?k():(r.Deferred.getStackHook&&(k.stackTrace=r.Deferred.getStackHook()),a.setTimeout(k))}}return r.Deferred(function(a){c[0][3].add(g(0,a,r.isFunction(e)?e:N,a.notifyWith)),c[1][3].add(g(0,a,r.isFunction(b)?b:N)),c[2][3].add(g(0,a,r.isFunction(d)?d:O))}).promise()},promise:function(a){return null!=a?r.extend(a,e):e}},f={};return r.each(c,function(a,b){var g=b[2],h=b[5];e[b[1]]=g.add,h&&g.add(function(){d=h},c[3-a][2].disable,c[0][2].lock),g.add(b[3].fire),f[b[0]]=function(){return f[b[0]+"With"](this===f?void 0:this,arguments),this},f[b[0]+"With"]=g.fireWith}),e.promise(f),b&&b.call(f,f),f},when:function(a){var b=arguments.length,c=b,d=Array(c),e=f.call(arguments),g=r.Deferred(),h=function(a){return function(c){d[a]=this,e[a]=arguments.length>1?f.call(arguments):c,--b||g.resolveWith(d,e)}};if(b<=1&&(P(a,g.done(h(c)).resolve,g.reject,!b),"pending"===g.state()||r.isFunction(e[c]&&e[c].then)))return g.then();while(c--)P(e[c],h(c),g.reject);return g.promise()}});var Q=/^(Eval|Internal|Range|Reference|Syntax|Type|URI)Error$/;r.Deferred.exceptionHook=function(b,c){a.console&&a.console.warn&&b&&Q.test(b.name)&&a.console.warn("jQuery.Deferred exception: "+b.message,b.stack,c)},r.readyException=function(b){a.setTimeout(function(){throw b})};var R=r.Deferred();r.fn.ready=function(a){return R.then(a)["catch"](function(a){r.readyException(a)}),this},r.extend({isReady:!1,readyWait:1,ready:function(a){(a===!0?--r.readyWait:r.isReady)||(r.isReady=!0,a!==!0&&--r.readyWait>0||R.resolveWith(d,[r]))}}),r.ready.then=R.then;function S(){d.removeEventListener("DOMContentLoaded",S), -a.removeEventListener("load",S),r.ready()}"complete"===d.readyState||"loading"!==d.readyState&&!d.documentElement.doScroll?a.setTimeout(r.ready):(d.addEventListener("DOMContentLoaded",S),a.addEventListener("load",S));var T=function(a,b,c,d,e,f,g){var h=0,i=a.length,j=null==c;if("object"===r.type(c)){e=!0;for(h in c)T(a,b,h,c[h],!0,f,g)}else if(void 0!==d&&(e=!0,r.isFunction(d)||(g=!0),j&&(g?(b.call(a,d),b=null):(j=b,b=function(a,b,c){return j.call(r(a),c)})),b))for(;h1,null,!0)},removeData:function(a){return this.each(function(){X.remove(this,a)})}}),r.extend({queue:function(a,b,c){var d;if(a)return b=(b||"fx")+"queue",d=W.get(a,b),c&&(!d||Array.isArray(c)?d=W.access(a,b,r.makeArray(c)):d.push(c)),d||[]},dequeue:function(a,b){b=b||"fx";var c=r.queue(a,b),d=c.length,e=c.shift(),f=r._queueHooks(a,b),g=function(){r.dequeue(a,b)};"inprogress"===e&&(e=c.shift(),d--),e&&("fx"===b&&c.unshift("inprogress"),delete f.stop,e.call(a,g,f)),!d&&f&&f.empty.fire()},_queueHooks:function(a,b){var c=b+"queueHooks";return W.get(a,c)||W.access(a,c,{empty:r.Callbacks("once memory").add(function(){W.remove(a,[b+"queue",c])})})}}),r.fn.extend({queue:function(a,b){var c=2;return"string"!=typeof a&&(b=a,a="fx",c--),arguments.length\x20\t\r\n\f]+)/i,la=/^$|\/(?:java|ecma)script/i,ma={option:[1,""],thead:[1,"","
"],col:[2,"","
"],tr:[2,"","
"],td:[3,"","
"],_default:[0,"",""]};ma.optgroup=ma.option,ma.tbody=ma.tfoot=ma.colgroup=ma.caption=ma.thead,ma.th=ma.td;function na(a,b){var c;return c="undefined"!=typeof a.getElementsByTagName?a.getElementsByTagName(b||"*"):"undefined"!=typeof a.querySelectorAll?a.querySelectorAll(b||"*"):[],void 0===b||b&&B(a,b)?r.merge([a],c):c}function oa(a,b){for(var c=0,d=a.length;c-1)e&&e.push(f);else if(j=r.contains(f.ownerDocument,f),g=na(l.appendChild(f),"script"),j&&oa(g),c){k=0;while(f=g[k++])la.test(f.type||"")&&c.push(f)}return l}!function(){var a=d.createDocumentFragment(),b=a.appendChild(d.createElement("div")),c=d.createElement("input");c.setAttribute("type","radio"),c.setAttribute("checked","checked"),c.setAttribute("name","t"),b.appendChild(c),o.checkClone=b.cloneNode(!0).cloneNode(!0).lastChild.checked,b.innerHTML="",o.noCloneChecked=!!b.cloneNode(!0).lastChild.defaultValue}();var ra=d.documentElement,sa=/^key/,ta=/^(?:mouse|pointer|contextmenu|drag|drop)|click/,ua=/^([^.]*)(?:\.(.+)|)/;function va(){return!0}function wa(){return!1}function xa(){try{return d.activeElement}catch(a){}}function ya(a,b,c,d,e,f){var g,h;if("object"==typeof b){"string"!=typeof c&&(d=d||c,c=void 0);for(h in b)ya(a,h,c,d,b[h],f);return a}if(null==d&&null==e?(e=c,d=c=void 0):null==e&&("string"==typeof c?(e=d,d=void 0):(e=d,d=c,c=void 0)),e===!1)e=wa;else if(!e)return a;return 1===f&&(g=e,e=function(a){return r().off(a),g.apply(this,arguments)},e.guid=g.guid||(g.guid=r.guid++)),a.each(function(){r.event.add(this,b,e,d,c)})}r.event={global:{},add:function(a,b,c,d,e){var f,g,h,i,j,k,l,m,n,o,p,q=W.get(a);if(q){c.handler&&(f=c,c=f.handler,e=f.selector),e&&r.find.matchesSelector(ra,e),c.guid||(c.guid=r.guid++),(i=q.events)||(i=q.events={}),(g=q.handle)||(g=q.handle=function(b){return"undefined"!=typeof r&&r.event.triggered!==b.type?r.event.dispatch.apply(a,arguments):void 0}),b=(b||"").match(L)||[""],j=b.length;while(j--)h=ua.exec(b[j])||[],n=p=h[1],o=(h[2]||"").split(".").sort(),n&&(l=r.event.special[n]||{},n=(e?l.delegateType:l.bindType)||n,l=r.event.special[n]||{},k=r.extend({type:n,origType:p,data:d,handler:c,guid:c.guid,selector:e,needsContext:e&&r.expr.match.needsContext.test(e),namespace:o.join(".")},f),(m=i[n])||(m=i[n]=[],m.delegateCount=0,l.setup&&l.setup.call(a,d,o,g)!==!1||a.addEventListener&&a.addEventListener(n,g)),l.add&&(l.add.call(a,k),k.handler.guid||(k.handler.guid=c.guid)),e?m.splice(m.delegateCount++,0,k):m.push(k),r.event.global[n]=!0)}},remove:function(a,b,c,d,e){var f,g,h,i,j,k,l,m,n,o,p,q=W.hasData(a)&&W.get(a);if(q&&(i=q.events)){b=(b||"").match(L)||[""],j=b.length;while(j--)if(h=ua.exec(b[j])||[],n=p=h[1],o=(h[2]||"").split(".").sort(),n){l=r.event.special[n]||{},n=(d?l.delegateType:l.bindType)||n,m=i[n]||[],h=h[2]&&new RegExp("(^|\\.)"+o.join("\\.(?:.*\\.|)")+"(\\.|$)"),g=f=m.length;while(f--)k=m[f],!e&&p!==k.origType||c&&c.guid!==k.guid||h&&!h.test(k.namespace)||d&&d!==k.selector&&("**"!==d||!k.selector)||(m.splice(f,1),k.selector&&m.delegateCount--,l.remove&&l.remove.call(a,k));g&&!m.length&&(l.teardown&&l.teardown.call(a,o,q.handle)!==!1||r.removeEvent(a,n,q.handle),delete i[n])}else for(n in i)r.event.remove(a,n+b[j],c,d,!0);r.isEmptyObject(i)&&W.remove(a,"handle events")}},dispatch:function(a){var b=r.event.fix(a),c,d,e,f,g,h,i=new Array(arguments.length),j=(W.get(this,"events")||{})[b.type]||[],k=r.event.special[b.type]||{};for(i[0]=b,c=1;c=1))for(;j!==this;j=j.parentNode||this)if(1===j.nodeType&&("click"!==a.type||j.disabled!==!0)){for(f=[],g={},c=0;c-1:r.find(e,this,null,[j]).length),g[e]&&f.push(d);f.length&&h.push({elem:j,handlers:f})}return j=this,i\x20\t\r\n\f]*)[^>]*)\/>/gi,Aa=/\s*$/g;function Ea(a,b){return B(a,"table")&&B(11!==b.nodeType?b:b.firstChild,"tr")?r(">tbody",a)[0]||a:a}function Fa(a){return a.type=(null!==a.getAttribute("type"))+"/"+a.type,a}function Ga(a){var b=Ca.exec(a.type);return b?a.type=b[1]:a.removeAttribute("type"),a}function Ha(a,b){var c,d,e,f,g,h,i,j;if(1===b.nodeType){if(W.hasData(a)&&(f=W.access(a),g=W.set(b,f),j=f.events)){delete g.handle,g.events={};for(e in j)for(c=0,d=j[e].length;c1&&"string"==typeof q&&!o.checkClone&&Ba.test(q))return a.each(function(e){var f=a.eq(e);s&&(b[0]=q.call(this,e,f.html())),Ja(f,b,c,d)});if(m&&(e=qa(b,a[0].ownerDocument,!1,a,d),f=e.firstChild,1===e.childNodes.length&&(e=f),f||d)){for(h=r.map(na(e,"script"),Fa),i=h.length;l")},clone:function(a,b,c){var d,e,f,g,h=a.cloneNode(!0),i=r.contains(a.ownerDocument,a);if(!(o.noCloneChecked||1!==a.nodeType&&11!==a.nodeType||r.isXMLDoc(a)))for(g=na(h),f=na(a),d=0,e=f.length;d0&&oa(g,!i&&na(a,"script")),h},cleanData:function(a){for(var b,c,d,e=r.event.special,f=0;void 0!==(c=a[f]);f++)if(U(c)){if(b=c[W.expando]){if(b.events)for(d in b.events)e[d]?r.event.remove(c,d):r.removeEvent(c,d,b.handle);c[W.expando]=void 0}c[X.expando]&&(c[X.expando]=void 0)}}}),r.fn.extend({detach:function(a){return Ka(this,a,!0)},remove:function(a){return Ka(this,a)},text:function(a){return T(this,function(a){return void 0===a?r.text(this):this.empty().each(function(){1!==this.nodeType&&11!==this.nodeType&&9!==this.nodeType||(this.textContent=a)})},null,a,arguments.length)},append:function(){return Ja(this,arguments,function(a){if(1===this.nodeType||11===this.nodeType||9===this.nodeType){var b=Ea(this,a);b.appendChild(a)}})},prepend:function(){return Ja(this,arguments,function(a){if(1===this.nodeType||11===this.nodeType||9===this.nodeType){var b=Ea(this,a);b.insertBefore(a,b.firstChild)}})},before:function(){return Ja(this,arguments,function(a){this.parentNode&&this.parentNode.insertBefore(a,this)})},after:function(){return Ja(this,arguments,function(a){this.parentNode&&this.parentNode.insertBefore(a,this.nextSibling)})},empty:function(){for(var a,b=0;null!=(a=this[b]);b++)1===a.nodeType&&(r.cleanData(na(a,!1)),a.textContent="");return this},clone:function(a,b){return a=null!=a&&a,b=null==b?a:b,this.map(function(){return r.clone(this,a,b)})},html:function(a){return T(this,function(a){var b=this[0]||{},c=0,d=this.length;if(void 0===a&&1===b.nodeType)return b.innerHTML;if("string"==typeof a&&!Aa.test(a)&&!ma[(ka.exec(a)||["",""])[1].toLowerCase()]){a=r.htmlPrefilter(a);try{for(;c1)}});function _a(a,b,c,d,e){return new _a.prototype.init(a,b,c,d,e)}r.Tween=_a,_a.prototype={constructor:_a,init:function(a,b,c,d,e,f){this.elem=a,this.prop=c,this.easing=e||r.easing._default,this.options=b,this.start=this.now=this.cur(),this.end=d,this.unit=f||(r.cssNumber[c]?"":"px")},cur:function(){var a=_a.propHooks[this.prop];return a&&a.get?a.get(this):_a.propHooks._default.get(this)},run:function(a){var b,c=_a.propHooks[this.prop];return this.options.duration?this.pos=b=r.easing[this.easing](a,this.options.duration*a,0,1,this.options.duration):this.pos=b=a,this.now=(this.end-this.start)*b+this.start,this.options.step&&this.options.step.call(this.elem,this.now,this),c&&c.set?c.set(this):_a.propHooks._default.set(this),this}},_a.prototype.init.prototype=_a.prototype,_a.propHooks={_default:{get:function(a){var b;return 1!==a.elem.nodeType||null!=a.elem[a.prop]&&null==a.elem.style[a.prop]?a.elem[a.prop]:(b=r.css(a.elem,a.prop,""),b&&"auto"!==b?b:0)},set:function(a){r.fx.step[a.prop]?r.fx.step[a.prop](a):1!==a.elem.nodeType||null==a.elem.style[r.cssProps[a.prop]]&&!r.cssHooks[a.prop]?a.elem[a.prop]=a.now:r.style(a.elem,a.prop,a.now+a.unit)}}},_a.propHooks.scrollTop=_a.propHooks.scrollLeft={set:function(a){a.elem.nodeType&&a.elem.parentNode&&(a.elem[a.prop]=a.now)}},r.easing={linear:function(a){return a},swing:function(a){return.5-Math.cos(a*Math.PI)/2},_default:"swing"},r.fx=_a.prototype.init,r.fx.step={};var ab,bb,cb=/^(?:toggle|show|hide)$/,db=/queueHooks$/;function eb(){bb&&(d.hidden===!1&&a.requestAnimationFrame?a.requestAnimationFrame(eb):a.setTimeout(eb,r.fx.interval),r.fx.tick())}function fb(){return a.setTimeout(function(){ab=void 0}),ab=r.now()}function gb(a,b){var c,d=0,e={height:a};for(b=b?1:0;d<4;d+=2-b)c=ca[d],e["margin"+c]=e["padding"+c]=a;return b&&(e.opacity=e.width=a),e}function hb(a,b,c){for(var d,e=(kb.tweeners[b]||[]).concat(kb.tweeners["*"]),f=0,g=e.length;f1)},removeAttr:function(a){return this.each(function(){r.removeAttr(this,a)})}}),r.extend({attr:function(a,b,c){var d,e,f=a.nodeType;if(3!==f&&8!==f&&2!==f)return"undefined"==typeof a.getAttribute?r.prop(a,b,c):(1===f&&r.isXMLDoc(a)||(e=r.attrHooks[b.toLowerCase()]||(r.expr.match.bool.test(b)?lb:void 0)),void 0!==c?null===c?void r.removeAttr(a,b):e&&"set"in e&&void 0!==(d=e.set(a,c,b))?d:(a.setAttribute(b,c+""),c):e&&"get"in e&&null!==(d=e.get(a,b))?d:(d=r.find.attr(a,b), -null==d?void 0:d))},attrHooks:{type:{set:function(a,b){if(!o.radioValue&&"radio"===b&&B(a,"input")){var c=a.value;return a.setAttribute("type",b),c&&(a.value=c),b}}}},removeAttr:function(a,b){var c,d=0,e=b&&b.match(L);if(e&&1===a.nodeType)while(c=e[d++])a.removeAttribute(c)}}),lb={set:function(a,b,c){return b===!1?r.removeAttr(a,c):a.setAttribute(c,c),c}},r.each(r.expr.match.bool.source.match(/\w+/g),function(a,b){var c=mb[b]||r.find.attr;mb[b]=function(a,b,d){var e,f,g=b.toLowerCase();return d||(f=mb[g],mb[g]=e,e=null!=c(a,b,d)?g:null,mb[g]=f),e}});var nb=/^(?:input|select|textarea|button)$/i,ob=/^(?:a|area)$/i;r.fn.extend({prop:function(a,b){return T(this,r.prop,a,b,arguments.length>1)},removeProp:function(a){return this.each(function(){delete this[r.propFix[a]||a]})}}),r.extend({prop:function(a,b,c){var d,e,f=a.nodeType;if(3!==f&&8!==f&&2!==f)return 1===f&&r.isXMLDoc(a)||(b=r.propFix[b]||b,e=r.propHooks[b]),void 0!==c?e&&"set"in e&&void 0!==(d=e.set(a,c,b))?d:a[b]=c:e&&"get"in e&&null!==(d=e.get(a,b))?d:a[b]},propHooks:{tabIndex:{get:function(a){var b=r.find.attr(a,"tabindex");return b?parseInt(b,10):nb.test(a.nodeName)||ob.test(a.nodeName)&&a.href?0:-1}}},propFix:{"for":"htmlFor","class":"className"}}),o.optSelected||(r.propHooks.selected={get:function(a){var b=a.parentNode;return b&&b.parentNode&&b.parentNode.selectedIndex,null},set:function(a){var b=a.parentNode;b&&(b.selectedIndex,b.parentNode&&b.parentNode.selectedIndex)}}),r.each(["tabIndex","readOnly","maxLength","cellSpacing","cellPadding","rowSpan","colSpan","useMap","frameBorder","contentEditable"],function(){r.propFix[this.toLowerCase()]=this});function pb(a){var b=a.match(L)||[];return b.join(" ")}function qb(a){return a.getAttribute&&a.getAttribute("class")||""}r.fn.extend({addClass:function(a){var b,c,d,e,f,g,h,i=0;if(r.isFunction(a))return this.each(function(b){r(this).addClass(a.call(this,b,qb(this)))});if("string"==typeof a&&a){b=a.match(L)||[];while(c=this[i++])if(e=qb(c),d=1===c.nodeType&&" "+pb(e)+" "){g=0;while(f=b[g++])d.indexOf(" "+f+" ")<0&&(d+=f+" ");h=pb(d),e!==h&&c.setAttribute("class",h)}}return this},removeClass:function(a){var b,c,d,e,f,g,h,i=0;if(r.isFunction(a))return this.each(function(b){r(this).removeClass(a.call(this,b,qb(this)))});if(!arguments.length)return this.attr("class","");if("string"==typeof a&&a){b=a.match(L)||[];while(c=this[i++])if(e=qb(c),d=1===c.nodeType&&" "+pb(e)+" "){g=0;while(f=b[g++])while(d.indexOf(" "+f+" ")>-1)d=d.replace(" "+f+" "," ");h=pb(d),e!==h&&c.setAttribute("class",h)}}return this},toggleClass:function(a,b){var c=typeof a;return"boolean"==typeof b&&"string"===c?b?this.addClass(a):this.removeClass(a):r.isFunction(a)?this.each(function(c){r(this).toggleClass(a.call(this,c,qb(this),b),b)}):this.each(function(){var b,d,e,f;if("string"===c){d=0,e=r(this),f=a.match(L)||[];while(b=f[d++])e.hasClass(b)?e.removeClass(b):e.addClass(b)}else void 0!==a&&"boolean"!==c||(b=qb(this),b&&W.set(this,"__className__",b),this.setAttribute&&this.setAttribute("class",b||a===!1?"":W.get(this,"__className__")||""))})},hasClass:function(a){var b,c,d=0;b=" "+a+" ";while(c=this[d++])if(1===c.nodeType&&(" "+pb(qb(c))+" ").indexOf(b)>-1)return!0;return!1}});var rb=/\r/g;r.fn.extend({val:function(a){var b,c,d,e=this[0];{if(arguments.length)return d=r.isFunction(a),this.each(function(c){var e;1===this.nodeType&&(e=d?a.call(this,c,r(this).val()):a,null==e?e="":"number"==typeof e?e+="":Array.isArray(e)&&(e=r.map(e,function(a){return null==a?"":a+""})),b=r.valHooks[this.type]||r.valHooks[this.nodeName.toLowerCase()],b&&"set"in b&&void 0!==b.set(this,e,"value")||(this.value=e))});if(e)return b=r.valHooks[e.type]||r.valHooks[e.nodeName.toLowerCase()],b&&"get"in b&&void 0!==(c=b.get(e,"value"))?c:(c=e.value,"string"==typeof c?c.replace(rb,""):null==c?"":c)}}}),r.extend({valHooks:{option:{get:function(a){var b=r.find.attr(a,"value");return null!=b?b:pb(r.text(a))}},select:{get:function(a){var b,c,d,e=a.options,f=a.selectedIndex,g="select-one"===a.type,h=g?null:[],i=g?f+1:e.length;for(d=f<0?i:g?f:0;d-1)&&(c=!0);return c||(a.selectedIndex=-1),f}}}}),r.each(["radio","checkbox"],function(){r.valHooks[this]={set:function(a,b){if(Array.isArray(b))return a.checked=r.inArray(r(a).val(),b)>-1}},o.checkOn||(r.valHooks[this].get=function(a){return null===a.getAttribute("value")?"on":a.value})});var sb=/^(?:focusinfocus|focusoutblur)$/;r.extend(r.event,{trigger:function(b,c,e,f){var g,h,i,j,k,m,n,o=[e||d],p=l.call(b,"type")?b.type:b,q=l.call(b,"namespace")?b.namespace.split("."):[];if(h=i=e=e||d,3!==e.nodeType&&8!==e.nodeType&&!sb.test(p+r.event.triggered)&&(p.indexOf(".")>-1&&(q=p.split("."),p=q.shift(),q.sort()),k=p.indexOf(":")<0&&"on"+p,b=b[r.expando]?b:new r.Event(p,"object"==typeof b&&b),b.isTrigger=f?2:3,b.namespace=q.join("."),b.rnamespace=b.namespace?new RegExp("(^|\\.)"+q.join("\\.(?:.*\\.|)")+"(\\.|$)"):null,b.result=void 0,b.target||(b.target=e),c=null==c?[b]:r.makeArray(c,[b]),n=r.event.special[p]||{},f||!n.trigger||n.trigger.apply(e,c)!==!1)){if(!f&&!n.noBubble&&!r.isWindow(e)){for(j=n.delegateType||p,sb.test(j+p)||(h=h.parentNode);h;h=h.parentNode)o.push(h),i=h;i===(e.ownerDocument||d)&&o.push(i.defaultView||i.parentWindow||a)}g=0;while((h=o[g++])&&!b.isPropagationStopped())b.type=g>1?j:n.bindType||p,m=(W.get(h,"events")||{})[b.type]&&W.get(h,"handle"),m&&m.apply(h,c),m=k&&h[k],m&&m.apply&&U(h)&&(b.result=m.apply(h,c),b.result===!1&&b.preventDefault());return b.type=p,f||b.isDefaultPrevented()||n._default&&n._default.apply(o.pop(),c)!==!1||!U(e)||k&&r.isFunction(e[p])&&!r.isWindow(e)&&(i=e[k],i&&(e[k]=null),r.event.triggered=p,e[p](),r.event.triggered=void 0,i&&(e[k]=i)),b.result}},simulate:function(a,b,c){var d=r.extend(new r.Event,c,{type:a,isSimulated:!0});r.event.trigger(d,null,b)}}),r.fn.extend({trigger:function(a,b){return this.each(function(){r.event.trigger(a,b,this)})},triggerHandler:function(a,b){var c=this[0];if(c)return r.event.trigger(a,b,c,!0)}}),r.each("blur focus focusin focusout resize scroll click dblclick mousedown mouseup mousemove mouseover mouseout mouseenter mouseleave change select submit keydown keypress keyup contextmenu".split(" "),function(a,b){r.fn[b]=function(a,c){return arguments.length>0?this.on(b,null,a,c):this.trigger(b)}}),r.fn.extend({hover:function(a,b){return this.mouseenter(a).mouseleave(b||a)}}),o.focusin="onfocusin"in a,o.focusin||r.each({focus:"focusin",blur:"focusout"},function(a,b){var c=function(a){r.event.simulate(b,a.target,r.event.fix(a))};r.event.special[b]={setup:function(){var d=this.ownerDocument||this,e=W.access(d,b);e||d.addEventListener(a,c,!0),W.access(d,b,(e||0)+1)},teardown:function(){var d=this.ownerDocument||this,e=W.access(d,b)-1;e?W.access(d,b,e):(d.removeEventListener(a,c,!0),W.remove(d,b))}}});var tb=a.location,ub=r.now(),vb=/\?/;r.parseXML=function(b){var c;if(!b||"string"!=typeof b)return null;try{c=(new a.DOMParser).parseFromString(b,"text/xml")}catch(d){c=void 0}return c&&!c.getElementsByTagName("parsererror").length||r.error("Invalid XML: "+b),c};var wb=/\[\]$/,xb=/\r?\n/g,yb=/^(?:submit|button|image|reset|file)$/i,zb=/^(?:input|select|textarea|keygen)/i;function Ab(a,b,c,d){var e;if(Array.isArray(b))r.each(b,function(b,e){c||wb.test(a)?d(a,e):Ab(a+"["+("object"==typeof e&&null!=e?b:"")+"]",e,c,d)});else if(c||"object"!==r.type(b))d(a,b);else for(e in b)Ab(a+"["+e+"]",b[e],c,d)}r.param=function(a,b){var c,d=[],e=function(a,b){var c=r.isFunction(b)?b():b;d[d.length]=encodeURIComponent(a)+"="+encodeURIComponent(null==c?"":c)};if(Array.isArray(a)||a.jquery&&!r.isPlainObject(a))r.each(a,function(){e(this.name,this.value)});else for(c in a)Ab(c,a[c],b,e);return d.join("&")},r.fn.extend({serialize:function(){return r.param(this.serializeArray())},serializeArray:function(){return this.map(function(){var a=r.prop(this,"elements");return a?r.makeArray(a):this}).filter(function(){var a=this.type;return this.name&&!r(this).is(":disabled")&&zb.test(this.nodeName)&&!yb.test(a)&&(this.checked||!ja.test(a))}).map(function(a,b){var c=r(this).val();return null==c?null:Array.isArray(c)?r.map(c,function(a){return{name:b.name,value:a.replace(xb,"\r\n")}}):{name:b.name,value:c.replace(xb,"\r\n")}}).get()}});var Bb=/%20/g,Cb=/#.*$/,Db=/([?&])_=[^&]*/,Eb=/^(.*?):[ \t]*([^\r\n]*)$/gm,Fb=/^(?:about|app|app-storage|.+-extension|file|res|widget):$/,Gb=/^(?:GET|HEAD)$/,Hb=/^\/\//,Ib={},Jb={},Kb="*/".concat("*"),Lb=d.createElement("a");Lb.href=tb.href;function Mb(a){return function(b,c){"string"!=typeof b&&(c=b,b="*");var d,e=0,f=b.toLowerCase().match(L)||[];if(r.isFunction(c))while(d=f[e++])"+"===d[0]?(d=d.slice(1)||"*",(a[d]=a[d]||[]).unshift(c)):(a[d]=a[d]||[]).push(c)}}function Nb(a,b,c,d){var e={},f=a===Jb;function g(h){var i;return e[h]=!0,r.each(a[h]||[],function(a,h){var j=h(b,c,d);return"string"!=typeof j||f||e[j]?f?!(i=j):void 0:(b.dataTypes.unshift(j),g(j),!1)}),i}return g(b.dataTypes[0])||!e["*"]&&g("*")}function Ob(a,b){var c,d,e=r.ajaxSettings.flatOptions||{};for(c in b)void 0!==b[c]&&((e[c]?a:d||(d={}))[c]=b[c]);return d&&r.extend(!0,a,d),a}function Pb(a,b,c){var d,e,f,g,h=a.contents,i=a.dataTypes;while("*"===i[0])i.shift(),void 0===d&&(d=a.mimeType||b.getResponseHeader("Content-Type"));if(d)for(e in h)if(h[e]&&h[e].test(d)){i.unshift(e);break}if(i[0]in c)f=i[0];else{for(e in c){if(!i[0]||a.converters[e+" "+i[0]]){f=e;break}g||(g=e)}f=f||g}if(f)return f!==i[0]&&i.unshift(f),c[f]}function Qb(a,b,c,d){var e,f,g,h,i,j={},k=a.dataTypes.slice();if(k[1])for(g in a.converters)j[g.toLowerCase()]=a.converters[g];f=k.shift();while(f)if(a.responseFields[f]&&(c[a.responseFields[f]]=b),!i&&d&&a.dataFilter&&(b=a.dataFilter(b,a.dataType)),i=f,f=k.shift())if("*"===f)f=i;else if("*"!==i&&i!==f){if(g=j[i+" "+f]||j["* "+f],!g)for(e in j)if(h=e.split(" "),h[1]===f&&(g=j[i+" "+h[0]]||j["* "+h[0]])){g===!0?g=j[e]:j[e]!==!0&&(f=h[0],k.unshift(h[1]));break}if(g!==!0)if(g&&a["throws"])b=g(b);else try{b=g(b)}catch(l){return{state:"parsererror",error:g?l:"No conversion from "+i+" to "+f}}}return{state:"success",data:b}}r.extend({active:0,lastModified:{},etag:{},ajaxSettings:{url:tb.href,type:"GET",isLocal:Fb.test(tb.protocol),global:!0,processData:!0,async:!0,contentType:"application/x-www-form-urlencoded; charset=UTF-8",accepts:{"*":Kb,text:"text/plain",html:"text/html",xml:"application/xml, text/xml",json:"application/json, text/javascript"},contents:{xml:/\bxml\b/,html:/\bhtml/,json:/\bjson\b/},responseFields:{xml:"responseXML",text:"responseText",json:"responseJSON"},converters:{"* text":String,"text html":!0,"text json":JSON.parse,"text xml":r.parseXML},flatOptions:{url:!0,context:!0}},ajaxSetup:function(a,b){return b?Ob(Ob(a,r.ajaxSettings),b):Ob(r.ajaxSettings,a)},ajaxPrefilter:Mb(Ib),ajaxTransport:Mb(Jb),ajax:function(b,c){"object"==typeof b&&(c=b,b=void 0),c=c||{};var e,f,g,h,i,j,k,l,m,n,o=r.ajaxSetup({},c),p=o.context||o,q=o.context&&(p.nodeType||p.jquery)?r(p):r.event,s=r.Deferred(),t=r.Callbacks("once memory"),u=o.statusCode||{},v={},w={},x="canceled",y={readyState:0,getResponseHeader:function(a){var b;if(k){if(!h){h={};while(b=Eb.exec(g))h[b[1].toLowerCase()]=b[2]}b=h[a.toLowerCase()]}return null==b?null:b},getAllResponseHeaders:function(){return k?g:null},setRequestHeader:function(a,b){return null==k&&(a=w[a.toLowerCase()]=w[a.toLowerCase()]||a,v[a]=b),this},overrideMimeType:function(a){return null==k&&(o.mimeType=a),this},statusCode:function(a){var b;if(a)if(k)y.always(a[y.status]);else for(b in a)u[b]=[u[b],a[b]];return this},abort:function(a){var b=a||x;return e&&e.abort(b),A(0,b),this}};if(s.promise(y),o.url=((b||o.url||tb.href)+"").replace(Hb,tb.protocol+"//"),o.type=c.method||c.type||o.method||o.type,o.dataTypes=(o.dataType||"*").toLowerCase().match(L)||[""],null==o.crossDomain){j=d.createElement("a");try{j.href=o.url,j.href=j.href,o.crossDomain=Lb.protocol+"//"+Lb.host!=j.protocol+"//"+j.host}catch(z){o.crossDomain=!0}}if(o.data&&o.processData&&"string"!=typeof o.data&&(o.data=r.param(o.data,o.traditional)),Nb(Ib,o,c,y),k)return y;l=r.event&&o.global,l&&0===r.active++&&r.event.trigger("ajaxStart"),o.type=o.type.toUpperCase(),o.hasContent=!Gb.test(o.type),f=o.url.replace(Cb,""),o.hasContent?o.data&&o.processData&&0===(o.contentType||"").indexOf("application/x-www-form-urlencoded")&&(o.data=o.data.replace(Bb,"+")):(n=o.url.slice(f.length),o.data&&(f+=(vb.test(f)?"&":"?")+o.data,delete o.data),o.cache===!1&&(f=f.replace(Db,"$1"),n=(vb.test(f)?"&":"?")+"_="+ub++ +n),o.url=f+n),o.ifModified&&(r.lastModified[f]&&y.setRequestHeader("If-Modified-Since",r.lastModified[f]),r.etag[f]&&y.setRequestHeader("If-None-Match",r.etag[f])),(o.data&&o.hasContent&&o.contentType!==!1||c.contentType)&&y.setRequestHeader("Content-Type",o.contentType),y.setRequestHeader("Accept",o.dataTypes[0]&&o.accepts[o.dataTypes[0]]?o.accepts[o.dataTypes[0]]+("*"!==o.dataTypes[0]?", "+Kb+"; q=0.01":""):o.accepts["*"]);for(m in o.headers)y.setRequestHeader(m,o.headers[m]);if(o.beforeSend&&(o.beforeSend.call(p,y,o)===!1||k))return y.abort();if(x="abort",t.add(o.complete),y.done(o.success),y.fail(o.error),e=Nb(Jb,o,c,y)){if(y.readyState=1,l&&q.trigger("ajaxSend",[y,o]),k)return y;o.async&&o.timeout>0&&(i=a.setTimeout(function(){y.abort("timeout")},o.timeout));try{k=!1,e.send(v,A)}catch(z){if(k)throw z;A(-1,z)}}else A(-1,"No Transport");function A(b,c,d,h){var j,m,n,v,w,x=c;k||(k=!0,i&&a.clearTimeout(i),e=void 0,g=h||"",y.readyState=b>0?4:0,j=b>=200&&b<300||304===b,d&&(v=Pb(o,y,d)),v=Qb(o,v,y,j),j?(o.ifModified&&(w=y.getResponseHeader("Last-Modified"),w&&(r.lastModified[f]=w),w=y.getResponseHeader("etag"),w&&(r.etag[f]=w)),204===b||"HEAD"===o.type?x="nocontent":304===b?x="notmodified":(x=v.state,m=v.data,n=v.error,j=!n)):(n=x,!b&&x||(x="error",b<0&&(b=0))),y.status=b,y.statusText=(c||x)+"",j?s.resolveWith(p,[m,x,y]):s.rejectWith(p,[y,x,n]),y.statusCode(u),u=void 0,l&&q.trigger(j?"ajaxSuccess":"ajaxError",[y,o,j?m:n]),t.fireWith(p,[y,x]),l&&(q.trigger("ajaxComplete",[y,o]),--r.active||r.event.trigger("ajaxStop")))}return y},getJSON:function(a,b,c){return r.get(a,b,c,"json")},getScript:function(a,b){return r.get(a,void 0,b,"script")}}),r.each(["get","post"],function(a,b){r[b]=function(a,c,d,e){return r.isFunction(c)&&(e=e||d,d=c,c=void 0),r.ajax(r.extend({url:a,type:b,dataType:e,data:c,success:d},r.isPlainObject(a)&&a))}}),r._evalUrl=function(a){return r.ajax({url:a,type:"GET",dataType:"script",cache:!0,async:!1,global:!1,"throws":!0})},r.fn.extend({wrapAll:function(a){var b;return this[0]&&(r.isFunction(a)&&(a=a.call(this[0])),b=r(a,this[0].ownerDocument).eq(0).clone(!0),this[0].parentNode&&b.insertBefore(this[0]),b.map(function(){var a=this;while(a.firstElementChild)a=a.firstElementChild;return a}).append(this)),this},wrapInner:function(a){return r.isFunction(a)?this.each(function(b){r(this).wrapInner(a.call(this,b))}):this.each(function(){var b=r(this),c=b.contents();c.length?c.wrapAll(a):b.append(a)})},wrap:function(a){var b=r.isFunction(a);return this.each(function(c){r(this).wrapAll(b?a.call(this,c):a)})},unwrap:function(a){return this.parent(a).not("body").each(function(){r(this).replaceWith(this.childNodes)}),this}}),r.expr.pseudos.hidden=function(a){return!r.expr.pseudos.visible(a)},r.expr.pseudos.visible=function(a){return!!(a.offsetWidth||a.offsetHeight||a.getClientRects().length)},r.ajaxSettings.xhr=function(){try{return new a.XMLHttpRequest}catch(b){}};var Rb={0:200,1223:204},Sb=r.ajaxSettings.xhr();o.cors=!!Sb&&"withCredentials"in Sb,o.ajax=Sb=!!Sb,r.ajaxTransport(function(b){var c,d;if(o.cors||Sb&&!b.crossDomain)return{send:function(e,f){var g,h=b.xhr();if(h.open(b.type,b.url,b.async,b.username,b.password),b.xhrFields)for(g in b.xhrFields)h[g]=b.xhrFields[g];b.mimeType&&h.overrideMimeType&&h.overrideMimeType(b.mimeType),b.crossDomain||e["X-Requested-With"]||(e["X-Requested-With"]="XMLHttpRequest");for(g in e)h.setRequestHeader(g,e[g]);c=function(a){return function(){c&&(c=d=h.onload=h.onerror=h.onabort=h.onreadystatechange=null,"abort"===a?h.abort():"error"===a?"number"!=typeof h.status?f(0,"error"):f(h.status,h.statusText):f(Rb[h.status]||h.status,h.statusText,"text"!==(h.responseType||"text")||"string"!=typeof h.responseText?{binary:h.response}:{text:h.responseText},h.getAllResponseHeaders()))}},h.onload=c(),d=h.onerror=c("error"),void 0!==h.onabort?h.onabort=d:h.onreadystatechange=function(){4===h.readyState&&a.setTimeout(function(){c&&d()})},c=c("abort");try{h.send(b.hasContent&&b.data||null)}catch(i){if(c)throw i}},abort:function(){c&&c()}}}),r.ajaxPrefilter(function(a){a.crossDomain&&(a.contents.script=!1)}),r.ajaxSetup({accepts:{script:"text/javascript, application/javascript, application/ecmascript, application/x-ecmascript"},contents:{script:/\b(?:java|ecma)script\b/},converters:{"text script":function(a){return r.globalEval(a),a}}}),r.ajaxPrefilter("script",function(a){void 0===a.cache&&(a.cache=!1),a.crossDomain&&(a.type="GET")}),r.ajaxTransport("script",function(a){if(a.crossDomain){var b,c;return{send:function(e,f){b=r(" + + + + + + diff --git a/static/docs/reference/generated/kubernetes-api/v1.13/jquery.scrollTo.min.js b/static/docs/reference/generated/kubernetes-api/v1.13/jquery.scrollTo.min.js new file mode 100644 index 0000000000..65a020d92a --- /dev/null +++ b/static/docs/reference/generated/kubernetes-api/v1.13/jquery.scrollTo.min.js @@ -0,0 +1,7 @@ +/** + * Copyright (c) 2007-2015 Ariel Flesler - afleslergmailcom | http://flesler.blogspot.com + * Licensed under MIT + * @author Ariel Flesler + * @version 2.1.2 + */ +;(function(f){"use strict";"function"===typeof define&&define.amd?define(["jquery"],f):"undefined"!==typeof module&&module.exports?module.exports=f(require("jquery")):f(jQuery)})(function($){"use strict";function n(a){return!a.nodeName||-1!==$.inArray(a.nodeName.toLowerCase(),["iframe","#document","html","body"])}function h(a){return $.isFunction(a)||$.isPlainObject(a)?a:{top:a,left:a}}var p=$.scrollTo=function(a,d,b){return $(window).scrollTo(a,d,b)};p.defaults={axis:"xy",duration:0,limit:!0};$.fn.scrollTo=function(a,d,b){"object"=== typeof d&&(b=d,d=0);"function"===typeof b&&(b={onAfter:b});"max"===a&&(a=9E9);b=$.extend({},p.defaults,b);d=d||b.duration;var u=b.queue&&1=f[g]?0:Math.min(f[g],n));!a&&1= node.offset().top) { + activeElemToken = token; + } + } + if (!prevElemToken) { + getNavElemNode(activeElemToken).addClass('selected'); + prevElemToken = activeElemToken; + return; + } + if (activeElemToken !== prevElemToken) { + getNavElemNode(prevElemToken).removeClass('selected'); + getNavElemNode(activeElemToken).addClass('selected'); + prevElemToken = activeElemToken; + } + return activeElemToken; + } + + function getHeadingNode(token) { + return $('#' + token); + } + + function getNavNode(token) { + return $('#' + token + '-nav'); + } + + function getNavElemNode(token) { + return $('#sidebar-wrapper > ul a[href="#' + token + '"]'); + } + + function checkNodePositions(nodes, flatNodeMap, scrollPosition) { + var activeNode; + for (var i = 0; i < nodes.length; i++) { + var item = nodes[i]; + var node = flatNodeMap[item.section]; + var nodeTop = node.offset().top - 50; + if (scrollPosition >= nodeTop) { + activeNode = {token: item.section, node: node}; + + if (item.subsections) { + activeNode.subsections = item.subsections; + } + break; + } + } + return activeNode; + } + + function scrollToNav(token) { + setTimeout(function() { + var scrollPosition = $(window).scrollTop(); + var activeSectionTokens = scrollActions(scrollPosition); + var activeElemToken = checkActiveElement(flatToc, scrollPosition); + var navNode = $('#sidebar-wrapper > ul a[href="#' + token + '"]'); + $('#sidebar-wrapper').scrollTo(navNode, {duration: 'fast', axis: 'y'}); + }, 200); + } + + $(window).on('hashchange', function(event) { + var scrollPosition = $(window).scrollTop(); + var activeSectionTokens = scrollActions(scrollPosition); + var activeElemToken = checkActiveElement(flatToc, scrollPosition); + var scrollToken = activeSectionTokens.L2 ? activeSectionTokens.L2 : activeSectionTokens.L1; + scrollToNav(scrollToken); + var token = location.hash.slice(1); + }); + + var scrollPosition = $(window).scrollTop(); + scrollActions(scrollPosition); + checkActiveElement(flatToc, scrollPosition); + // TODO: prevent scroll on sidebar from propogating to window + $(window).on('scroll', function(event) { + var scrollPosition = $(window).scrollTop(); + var activeSectionTokens = scrollActions(scrollPosition); + var activeElemToken = checkActiveElement(flatToc, scrollPosition); + }); +}); \ No newline at end of file diff --git a/static/images/kubeadm/kubeadm-ha-topology-external-etcd.svg b/static/images/kubeadm/kubeadm-ha-topology-external-etcd.svg new file mode 100755 index 0000000000..3a1d6ee027 --- /dev/null +++ b/static/images/kubeadm/kubeadm-ha-topology-external-etcd.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/static/images/kubeadm/kubeadm-ha-topology-stacked-etcd.svg b/static/images/kubeadm/kubeadm-ha-topology-stacked-etcd.svg new file mode 100755 index 0000000000..c7c0f70137 --- /dev/null +++ b/static/images/kubeadm/kubeadm-ha-topology-stacked-etcd.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/update-imported-docs/reference.yml b/update-imported-docs/reference.yml index 6410f38e4c..e44b495f22 100644 --- a/update-imported-docs/reference.yml +++ b/update-imported-docs/reference.yml @@ -8,7 +8,7 @@ repos: cd $GOPATH git clone https://github.com/kubernetes/kubernetes.git src/k8s.io/kubernetes cd src/k8s.io/kubernetes - git checkout release-1.12 + git checkout release-1.13 make generated_files cp -L -R vendor $GOPATH/src rm -r vendor diff --git a/update-imported-docs/release.yml b/update-imported-docs/release.yml index 0da74c95bd..3d3bd8850b 100644 --- a/update-imported-docs/release.yml +++ b/update-imported-docs/release.yml @@ -4,5 +4,5 @@ repos: branch: master gen-absolute-links: true files: - - src: CHANGELOG-1.12.md + - src: CHANGELOG-1.13.md dst: content/en/docs/setup/release/notes.md