Hyperledger Fabric Deployment Using Helm Chart

Henry Zhang, Jiahao Chen, Hui Hu, Wenkai Yin

This blog post is to share the personal technical experience of the authors. It does not represent in any way the opinions of the authors’ employers.

* We assume readers are familiar with Helm, Docker, Kubernetes, and have knowledge of the Fabric’s architecture.

This post introduces an approach to deploy Fabric via Helm Chart. The Chart allows a user to flexibly configure the consensus algorithm (solo/Kafka) and the number of organizations and nodes of the Fabric network.

Our previous blogs describe how to deploy Fabric on Kubernetes (K8s), which were widely read and followed by the community members. Many readers were inspired by our posts e and were able to use the K8s to manage their Fabric cluster. If you are new to this domain, you can review our previous posts and understand the steps that are used in deploying Fabric on Kubernetes.

Helm Chart is used as a preferred way by more and more developers to deploy their applications on Kubernetes. It is growing as to be a de facto standard. The advantage of Helm is its highly automated and repeatable deployment. A single “Helm” command can perform a series of actions, while it takes a number of “kubectl” commands to achieve the same purpose.

Introduction of Helm

Helm is a package manager for Kubernetes. It simplifies the deployment and management of Kubernetes applications. Helm has three important concepts:

  • Chart: defines a package format that can be deployed on Kubernetes. A Chart contains a set of files that describe Kubernetes related resources.
  • Config: used to store the configuration information of the software, used together with Chart to create a Release.
  • Release: a running instance of a Chart.

Helm consists of two important functional components: the Helm Client and the Tiller Server.

  • Helm Client is a command-line tool for end users. It is used primarily for developing local charts, managing Chart repositories, and interacting with Tiller Server.
  • Tiller Server is installed in the Kubernetes cluster. It accepts requests from the Helm Client and interacts with the Kubernetes API Server, including responding to Helm Client requests, combining Chart and Config to create Releases, installing Charts into Kubernetes and keeping track of status.

Deploying Fabric by Helm Chart

  1. Helm installation

Please refer to the official document: https://docs.helm.sh/using_helm/#installing-helm

  1. Preparing the NFS server

In our deployment model, an NFS server is a place for sharing the configuration information among the Fabric nodes, so an available NFS server is required. For the details of how NFS server works in deployment, please refer to this post: How to Deploy Hyperledger Fabric on Kubernetes (1)

  1. Generating Certificates

In Fabric’ github repo (https://github.com/hyperledger/fabric), you could find a tool called “cryptogen”, which allows user to generate certificate files. The certificates are used by entities of Fabric network such as users and nodes.

In general, a user can define the organization structure in a YAML file, such as the number of organizations, the name and domain of organizations. The “cryptogen” tool can load the configuration file and generate corresponding certificate files.

A sample configuration file called “cluster-config.yaml” is as follows:


- Name: Orderer

Domain: example.com


Count: 1


- Name: Org1

Domain: org1.example.com


Count: 1

- Name: Org2

Domain: org2.example.com


Count: 1

The keywords of OrdererOrgs and PeerOrgs are used to distinguish the types of organizations. The structure of the two organizations is as follows:

  • OrdererOrgs defines the information of the orderer organization. The above sample defines an organization named “Orderer” whose domain name is “example.com”, and it specifies the value of “Template.Count” as 1, thus there is only one orderer certificate will be generated, which is identified by “orderer0”.
  • PeerOrgs defines the information of the peer organization. The above sample defines two organizations: “Org1” and “Org2” . The corresponding domain names are “org1.example.com” and “org2.example.com”. Like the orderer, a Peer is provisioned to each organization. The first peer in “Org1” and “Org2” are both identified as “peer0”. They can be easily distinguished by the domain name.

For more configuration information about cryptogen, please refer to the description in the source code.

  1. Downloading the Chart

Download the chart via git clone:




After downloading, go to the fabric -chart/ directory. 

  1. Editing “values.yaml” of the chart
  • Select the consensus algorithm: “consensusType” is used to configure consensus algorithm: solo or Kafka. Helm will render the deployment templates accordingly.
  • Modify the definition of organization: since the “crypto-config” directory generated in step 3 contains an hierarchy of subdirectories based on the file “cluster-config.yaml”, the same information needs to be included into “values.yaml” to ensure that Fabric nodes could find their certificate files correctly. The easiest way is to copy the content of the above “cluster-config.yaml” to “values.yaml”.
  • Configure the NFS server, add NFS server information to let Fabric nodes load their configuration files in the Kubernetes environment.

A sample “values.yaml” is as follows:

clusterName: mycluster

consensusType: solo



basePath: /opt/share


- name: orderer

domain: example.com


count: 1


- name: org1

domain: org1.example.com


count: 1

- name: org2

domain: org2.example.com


count: 1


  1. Moving the certificates to NFS server

The certificate files generated in step 3 are stored in a directory called “crypto-config/”. To allow Fabric nodes to obtain the certificate information, the certificate files should be copied onto the NFS server so that the nodes can access the certificate files through PV (persistent volume).

For example, assuming that the directory exported by the NFS server is “/opt/share”, “crypto-config” should be placed in the following path:

“/opt/share/<clusterName>/resources/crypto-config”, where clusterName is used to distinguish different instances of Fabric cluster. In our sample values.yaml in step 5, the clusterName is mycluster.

  1. Deploying Chart to Kubernetes

After the above steps, use the following command to deploy the Fabric to the Kubernetes cluster:

$ helm install .

The architecture of Fabric on Kubernetes is as follows:

  1. Installation principles

Basically, the Fabric Chart contains a set of dynamitic deployment templates. Before the actual deployment of Fabric, Helm renders the Fabric Chart based on “value.yaml”. It then pushes the rendered deployment files to Tiller server,   Tiller calls the Kubernetes APIs and finishes the deployment eventually.

We take the template file (fabric-chart/templates/peer.yaml) of the Peer node as an example to briefly explain this process:


{{- range $peerOrg := $root.Values.peerOrgs}}

In the definition of the template, each Org corresponds to a namespace in Kubernetes. Each peer is mapped to a Pod under its org’s namespace, therefore the rendering will iterate through all the Orgs.


{{- $namespace := printf "%s-%s" $orgName $clusterName }}

{{- $scope := dict "name" $namespace }}

{{- template "namespace" $scope }}

Render the namespace for an Org in each iteration. The namespace takes the format of “org.name-clustName”.


{{- $name := printf "%s-shared" $namespace }}

{{- $scope := dict "name" $name "nfsServer" $root.Values.nfs.ip "nfsPath" $nfsPath "pvcNamespace" $namespace "pvcName" $sharedPVCName }}

{{- template "persistentVolume" $scope }}

The same as the iteration described above, render the Persistent Volume and Persistent Volume Claim for Pods within the same namespace.


{{- range $index := until ($peerOrg.template.count | int) }}

{{- $name := printf "peer%d" $index }}

{{- $scope := dict "name" $name "namespace" $namespace "orgName" $orgName "orgDomainName" $orgDomainName "pvc" $sharedPVCName }}

{{- template "peer.deployment" $scope }}

{{- $name := printf "peer%d" $index }}

{{- $scope := dict "name" $name "namespace" $namespace }}

{{- template "peer.service" $scope }}

Finally, the deployment file of the Peer node will be rendered according to the Organization’s information. The deployment file is consisted of Kubernetes resources, such as deployment, service and ingress.

The templates also define other Fabric components like Orderer, CA and CLI. Readers can find them in fabric-chart/templates.

Related blogs:

Deploy Hyperledger Fabric on Kubernetes Part 1

Deploy Hyperledger Fabric on Kubernetes Part 2

Related Information:

A tool (including Blockchain Explorer) to automate the deployment Fabric on Kubernetes can be found at here.

Project Harbor Reached Milestone of 2000 Stars on Github

About a year ago, I gave the first star to an open source project we created. In less than 13 months, the project has reached an exciting milestone of 2000 stars!  This project is called Harbor, an enterprise class registry server.

People from different countries starred Project Harbor on Github

Back in early 2014, when I attended Docker meetups and container conferences, I often heard people complaining about the challenges to manage container images. They usually created all kinds of hacks or workarounds to solve their own problems. When I saw pain points like these, my gut feeling told me that it must be a great opportunity to do something to it.

We then started to work on a side project to help people manage image effectively. This project became the prototype of Harbor. It was used by a few project teams and turned out to be quite helpful. In March 2016, we decided to open source it on Github for larger adoption. Since then, Project Harbor has taken off and been gaining more and more traction. We listened to feedback from the community and kept improving it. Community developers were enthusiastic and they contributed code, tools, documentation and even translation to multiple languages to the project. Two third of the contributors was actually from outside of VMware.

Gradually, Harbor becomes one of the most popular open source registries and has been widely used by people in the container space. VMware has also integrated Harbor into two products: vSphere Integrated Containers and Photon Platform. Many users run Harbor in their production, such as one of the largest internet companies, JD.com in China. Other companies also forked Harbor and used in their own products. Below are some statistics of Project Harbor.

Project Harbor Statistics

The current version of Harbor provides some important features to enterprise users, such as RBAC (Role Based Access Control), LDAP/AD authentication, image remote replication, management portal. In the coming new release, Harbor will be adding new features like Notary and a new admin UI.

One of my favorite features of Harbor: Remote replication (synchronization) of images

While we are celebrating the milestone of Harbor, it certainly serves as a new starting point to us. Thanks everyone who contributed to Harbor’s success. Your continuous support definitely motivates us to make Harbor the best home for your container images!

Survey based on user community, 53 responses

Related Topics:

Architecture of Harbor: An Open Source Enterprise-class Registry Server
Private Docker Registry Harbor Achieves HA based on Virtual SAN
Working with Harbor Registry REST API via Swagger

Private Docker Registry Harbor Achieves HA based on Virtual SAN

Recently, VMware released the Docker Volume Driver for vSphere 1.0 beta, which enabled a Docker host to create volumes directly on a vSphere datastore (Virtual SAN, VMFS, NFS, etc). The volumes can be directly mounted into Docker containers. The Docker volume solves the problem of storing persistent data of Docker containers. The Docker Volume of vSphere not only simplifies storage configuration, the volumes can also be associated with the Storage Policy Based Management (SPBM) of vSphere. For example, an administrator can set Fault To Tolerant (FTT) or Stripe Width (SW) of the data volume. Volumes with SPBM can achieve a higher data protection level and better performance. The docker volume driver of vSphere is an open-source project. It is downloadable at https://github.com/vmware/docker-volume-vsphere .

This blog walks through the steps of creating data volumes in VMware Virtual SAN (VSAN). As an example of a containerized application, the open source Harbor Registry is used to describe the usage of data volumes provisioned by VSAN, through which Harbor Registry achieves a higher data protection level and high availability (HA).

A little more background about Harbor Registry: it is another open-source project by VMware. A registry is one of the necessary components of a container’s build-ship-run lifecycle. Harbor helps users set up an enterprise private Docker registry service rapidly. Furthermore, it also provides enhanced features usually required by enterprises such as graphical user interface (GUI), role based access control, AD/LDAP integration and image replication. Harbor’s Github repo: https://github.com/vmware/harbor .

vsanharborha1The architecture of the system is illustrated in the above figure. 3 ESXi hosts form a VSAN cluster. A Harbor registry VM is running on one of the hosts. Besides, there are three external Docker volumes created in the VSAN cluster, used for storing persistent data in Harbor. This cluster provides consolidated storage by local disks of each host. It can tolerate a failure of one physical host and still preserve data integrity and accessibility.

The configuration process is discussed as follows.
1.    First, set up a Virtual SAN cluster with 3 ESXi hosts. A photon OS VM ( https://vmware.github.io/photon/ ) is installed on one of the ESXi server as a Docker host. Of course, other Linux distributions like Ubuntu can be used as well, as long as it can run Docker Engine and Docker Compose.

t12.    On the release page of Docker Volume Driver for vSphere project (https://github.com/vmware/docker-volume-vsphere/releases), download the plugin for ESXi host and for VMs respectively. For example, for 1.0 beta, the file names are:


3.    On each of the ESXi hosts, use the following commands to install the plugin (SSH of ESXi host must be enabled). After installation, no reboot is required.

# esxcli software vib install -d "/vmware-esx-vmdkops-1.0.beta.zip" \
--no-sig-check –f
Installation Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed: VMWare_bootbank_esx-vmdkops-service_1.0.0-0.0.1
VIBs Removed:
VIBs Skipped:

4.    On the Photon VM, install the RPM package. For other Debian based OS, install the corresponding deb package.

# rpm -ivh docker-volume-vsphere-1.0.beta-1.x86_64.rpm
Preparing...                              ##################### [100%]
Updating / installing...
1:docker-volume-vsphere-0:1.0.beta-############################ [100%]
File: '/proc/1/exe' -> '/usr/lib/systemd/systemd'
Created symlink from /etc/systemd/system/multi-user.target.wants/\
docker-volume-vsphere.service to /usr/lib/systemd/system/

5.    After the ESXi plugin is installed, a management script is generated at /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py. This script helps administrators manage the data volumes. For example, an administrator can create different storage policies. In Virtual SAN, the default storage policy has a Stripe Width setting of 1 (SW=1). We will create a new policy with SW=2 as an example.
To do this, just SSH into any of the ESXi hosts and run this command:

# /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py policy \
create --name SW=2 --content '(("stripeWidth" i2))'

The parameter ‘SW=2’ is the name of the policy. The key point here is to set the content of the policy and it is ‘((“stripeWidth” i2))’ in this example. Other settings are the same as the Virtual SAN policy parameters. The possible parameters and their description are as follows:spbm6.    Now Docker volumes can be created on the Docker host (the Photon OS VM). As an example, we first create two volumes with default storage policy and then create another volume with the newly created ‘SW=2’ policy.

# docker volume create --driver=vmdk --name=vsanvol1 -o size=50gb
# docker volume create --driver=vmdk --name=vsanvol2 -o size=20gb
# docker volume create --driver=vmdk --name=vsanvol3 -o size=20gb \
-o vsan-policy-name=SW=2

By specifying the ‘–driver=vmdk’ parameter, the external volume is created in the vSphere datastore. The volume is created in the same datastore where the Photon OS VM resides. In this example the Photon OS VM is stored in Virtual SAN, so are the Docker volumes. These volumes are stored in the form of VMDK. What is noteworthy here is that the volumes are not mounted to any VM by now. So if we navigate to the vSphere Web Client, we cannot find any information about these newly created volumes from the VM’s page.t2However, we can indeed find them in the dockvols directory in the Virtual SAN datastore.t3In subsequent sections, we are able to find the VMDKs through the VM’s page when the volumes are mounted to running containers.

7.    On the Photon OS VM, download the Harbor Registry source code. Before installing Harbor, we need to modify the harbor/Deploy/docker-compose.yml configuration file in order to use the newly created external volumes. We can then install Harbor by following the official Harbor installation guide.

Open the docker-compose.yml file. Find the ‘registry’ section, modify these lines:

  - /data/registry:/storage
  - ./config/registry/:/etc/registry/


- vsanvol1:/storage
- ./config/registry/:/etc/registry/

vsanvol1 is the external volume we just created.
Next, look for the ‘mysql’ section and modify these lines:

  - /data/database:/var/lib/mysql


  - vsanvol2:/var/lib/mysql

Similarly, vsanvol2 is another volume we just created.
Next, look for the ‘jobservice’ section and modify these lines:

  - /data/job_logs:/var/log/jobs
  - ./config/jobservice/app.conf:/etc/jobservice/app.conf


  - vsanvol3:/var/log/jobs
  - ./config/jobservice/app.conf:/etc/jobservice/app.conf

Similarly, vsanvol3 is another volume we just created.
In the end of the file, add the following lines:

    external: true
    external: true
    external: true

These lines indicate that these volumes have already been created and do not need to be created by Docker again. Keep other configurations unchanged in the docker-compose.yml. Then install Harbor as the official guide and bring up Harbor registry service.

8.    After Harbor is running, we can check the vSphere Web Client and confirm that these 3 external volumes are indeed mounted to the Photon OS VM. They are mounted as ‘Hard Disk 2’,‘Hard Disk 3’ and ‘Hard Disk 4’ in the VM respectively. In this beta version, there seems some bugs about displaying storage policy. For example, the storage policies for these VMDKs are displayed as ‘None’ while we can see that ‘Hard Disk 3’ is created as ‘SW=2’ policy and the other two VMDKs are created with the default storage policy. The below screenshot shows a storage policy of ‘Hard Disk 4’:t5There may be a problem where Virtual SAN cannot identify the storage policy created by ‘” Docker Volume Driver for vSphere’ correctly. This problem should be solved in newer version.

9.    Let’s upload two images to test if there is any data loss when a host fails.t610.    Enable vSphere HA on this Virtual SAN cluster, with default HA settings. Then we identify that the Photon OS VM is on the ESXi host with IP address    Power off the physical host with IP address Wait for a while after HA restarts the VM and check the state of Photon OS VM.t8The VM has been restarted on another heathy host. The original external volumes are mounted to the restarted VM. Because a host of the VSAN cluster is powered off, for each VMDK there will be a component shown as ‘absent’. However, with the default storage policy Virtual SAN can tolerate a host’s failure, so the access to the data is still successful.
12.    After Photon VM is restarted, check the status of Harbor. All the services and containers are running as normal.t1013.    Check Harbor UI, the 2 images we uploaded before are still intact. This indicates that there is no data loss.t11When vSphere HA restarted the Harbor VM on another healthy host, all the containers of Harbor are also restarted. They are connected to the original same volumes as in the figure:vsanharborha2This blog introduces an example of achieving Harbor registry HA by leveraging Virtual SAN and vSphere HA. Since Harbor is a multi-container application, this approach can also be applied to other container-based applications.

Related posts:

Architecture of Harbor: An Open Source Enterprise-class Registry Server

Working with Harbor Registry REST API via Swagger

Working with Harbor Registry REST API via Swagger

Swagger is the most popular RESTful API tool, it contains an entire set of codes, editors, code generators etc, and can be used in API descriptions, definitions, generation and visualization etc. For details about Swagger, see http://www.swagger.io, where you can download its source code and integrate it with the project.

Harbor is an enterprise-class private registry server initiated by VMware(http://github.com/vmware/harbor). Harbor also offers RESTful API which provides easy integration with other container management platforms. This article describes how to use Swagger tools embedded in Harbor to test RESTful APIs.

First, let’s take a look at how Swagger creates descriptions and definitions for RESTful API. Swagger provides an online WYSIWYG editor at http://editor.swagger.io/, users can enter Swagger-compatible YAML or JSON input on the left pane of the editor, and the result of the input will be shown on the right pane. If there are any input errors, there are alerts with amendment recommendations for the user, it’s very convenient! Refer to http://swagger.io/specification/ for instructions on writing definition files that are compatible with Swagger. This editor also supports the download of completed YAML to the local system, or conversion to JSON format. It can even help us auto-generate a Mockup Server or client.

Swagger Embedded in Harbor

Core functions of Harbor are implemented through RESTful API. A set of API rules that can be visualized was documented in Swagger during the development process and is provided for users as part of the project.

The Harbor Project utilizes two methods to let users present or control RESTFul API with Swagger.

The first is the “static” method, which only uses Swagger as the tool for presentations and reviews. Users only have to locate the swagger.yaml file from the directory docs/ of Project Harbor , and through the editor, open, select all, copy, and paste into the code pane on the left of Swagger online editor. The right pane will display a visualization of the Harbor RESTful API document page for review and reference.
article3_image1The second method is the “dynamic” method, which involves deploying Swagger UI and Harbor REST services in the same Server. Users can use Swagger to control and test Harbor RESTful APIs. This method may change data in the database, so it is not suggested to be used in production systems. Deployment procedures are illustrated in the figure below:article3_image2Under the directory docs/ of Harbor Project source codes, there is a script file named prepare-swagger.sh, which can help users carry out “dynamic” deployment. The following provides instructions on related steps. For detailed information, please refer to the file docs/configure_swagger.md:

(1) Change the SERVER_IP value in the script file, set it to the IP address of the host system of currently deployed Harbor system, save changes and execute the script. The script will download the Swagger software package accordingly and decompress it to the directory of static resources of Harbor Project vendors; copy the swagger.yaml files under docs/ to the Harbor Project static resource directory resources/yaml; change/replace URL contents according to the SERVER_IP provided by the user.

(2) Switch to the Deploy directory, change the file named “docker-compose.yml”, mount the newly-added Swagger static resource directory onto Harbor UI Docker container through Volumes, letting SwaggerUI deploy together with Harbor UI after starting up, to provide external access.

(3) Use the docker-compose command to re-create Project Harbor, clear all content left on the server, restart the newly created Project Harbor image.

The figure below shows a screenshot of a deployed Swagger UI page.
RESTful API Authentication

When triggering Harbor RESTful API using Swagger UI, please be aware of “login status” issues, because some of API requires session information. There are two ways to configure a session.

Method 1: Open the UI with a browser (Note: Make sure that the IP address of the URL in the Harbor UI is the same as the value provided for SERVER_IP when deploying Swagger UI), complete the registration (if using for the first time) and login; then open a new (tab) in the same browser, enter the Swagger UI address below, this will ensure that HarborRESTful API is running when the user is logged in.


Method 2: Harbor RESTful API supports Basic Authentication mode. However, Swagger currently does not allow the input of usernames and passwords on its interface, so access becomes inconvenient. Those who are interested can follow this link https://github.com/swagger-api/swagger-ui and try to make Swagger accessible in Basic Authentication mode. Of course, the user can also use the below command to access API. In this way, the user does not have to log in to Harbor’s UI in order to test the API.

curl -u <username: password>

Related article:

Harbor Architecture Overview