Cinder NFS Driver on OpenStack

在grizzly版本當中，cinder.volume.drivers.nfs.NfsDriver所製作出來的volumes就如同其他driver所製作出的一樣可以執行attach/detach的operation，但目前無法執行snapshot的運作。

OpenStack Security Group設定

為了讓Security Group的相關設定都交給Quantum Service來代理(proxy)處理，所以必須在/etc/nova/nova.conf中做以下的設定，以避免與原來Nova Networking所處理的Security Group相衝突。



## security groups ##
firewall_driver = nova.virt.firewall.NoopFirewallDriver
security_group_api = quantum

OpenStack: nova.conf section

 Sections

Configuration options are grouped by section. The Compute config file supports the following sections.

[DEFAULT]

Almost all of the configuration options are organized into this section. If the documentation for a configuration option does not specify its section, assume that it should be placed in this one.

[cells]

The cells section is used for options for configuring cells functionality. See the Cells section of the OpenStack Compute Admin Manual for more details.

[baremetal]

This section is used for options that relate to the baremetal hypervisor driver.

[conductor]

The conductor section is used for options for configuring the nova-conductor service.

[trusted_computing]

The trusted_computing section is used for options that relate to the trusted computing pools functionality. Options in this section describe how to connect to a remote attestation service.

Nova Conductor Service: no-db-compute for compute node

Q: 為何要做nova conductor service?這個從folsom版本開始討論構思，一直到了grizzly版本才合併。nova conductor究竟是啥樣子的服務呢!!?

[Dev List]
     Nova Conductor Dev List, http://lists.openstack.org/pipermail/openstack-dev/2012-November/002573.html

A:
主要是不透過Nova DB的存取來降低效能的瓶頸以及compromised compute node會帶來淺在的危險。

 The nova-conductor service is key to completing no-db-compute. Conceptually, it implements a new layer on top of nova-compute.



[Objective]
1. Compute nodes are the least trusted part of a nova deployment, so removing direct database access is a step toward reducing the potential impact of a compromised compute node.

2. The other benefit of no-db-compute is for upgrades. Direct database access complicates the ability to do live rolling upgrades

Reference:

1. http://cloudystuffhappens.blogspot.tw/2013/04/understanding-nova-conductor-in.html
2. http://russellbryantnet.wordpress.com/2013/02/19/deployment-considerations-for-nova-conductor-service-in-openstack-grizzly/

Nova rootwrap

使用root wrapper可以让非特权用户以root身份尽可能安全地执行部分操作。nova曾经使用sudoers文件来列出允许执行的特权命令，使用sudo来运行这么命令，但是这样不容易维护，而且不能进行复杂的参数处理，rootwrap就是为了解决这些问题。

使用sudo nova-rootwrap config-file command，而不再是使用sudo command。只需要使用一个通用的sudoers使nova-rootwrap以root身份运行。nova-rootwrap查看配置文件，加载command filters，检查请求的命令是否匹配某个filter，如果匹配就以root身份运行，否则就拒绝请求。

Reference:

1. http://my.oschina.net/chape/blog/119990

2. https://wiki.openstack.org/wiki/Nova/Rootwrap#How_rootwrap_works

Xenserver交換Networking Backend

Switching between Linux Bridge and Open VSwitch

Note: Open vSwitch has been the default network backend since XenServer 6.0.0 and XCP 1.5. If you switched to using Linux Bridge and this proved beneficial to you, please let us know.

To see what network backend you are currently using, run in dom0:

cat /etc/xensource/network.conf

To switch to using the Linux Bridge network backend, run in dom0:

xe-switch-network-backend bridge

To switch to using Open vSwitch network backend, run in dom0:

xe-switch-network-backend openvswitch

OpenvSwitch

* ovs-vswitchd, a daemon that implements the switch, along with 
      a companion Linux kernel module for flow-based switching.

* ovsdb-server, a lightweight database server that ovs-vswitchd
      queries to obtain its configuration.

* ovs-brcompatd, a daemon that allows ovs-vswitchd to act as a
      drop-in replacement for the Linux bridge in many environments, 
      along with a companion Linux kernel module to intercept bridge 
      ioctls.

* ovs-dpctl, a tool for configuring the switch kernel module.

* Scripts and specs for building RPMs for Citrix XenServer and Red
      Hat Enterprise Linux.  The XenServer RPMs allow Open vSwitch to
      be installed on a Citrix XenServer host as a drop-in replacement
      for its switch, with additional functionality.

* ovs-vsctl, a utility for querying and updating the configuration
      of ovs-vswitchd.

* ovs-appctl, a utility that sends commands to running Open
      vSwitch daemons.

* ovsdbmonitor, a GUI tool for remotely viewing OVS databases and
      OpenFlow flow tables.

安裝注意事項

Open vSwitch可以用kernel modules來support以降低cost，或者可以安裝到userspace但是會增加performance的cost。

OpenStack Nova: nova.conf 中的flag

https://github.com/openstack/nova/blob/stable/folsom/nova/flags.py

OpenStack spice vncproxy

Spice Console

緣由

OpenStack Compute has long had support for VNC consoles to guests. The VNC protocol is fairly limited, lacking support for multiple monitors, bi-directional audio, reliable cut+paste, video streaming and more. 

SPICE is a new protocol which aims to address all the limitations in VNC, to provide good remote desktop support.

SPICE support in OpenStack Compute shares a similar architecture to the VNC implementation. 

The OpenStack Dashboard uses a SPICE-HTML5 widget in its console tab, that communicates to the nova-spicehtml5proxy service using SPICE-over-websockets. The nova-spicehtml5proxy service communicates directly with the hypervisor process using SPICE.

Reference

1. http://en.wikipedia.org/wiki/SPICE_(protocol)

2. http://www.linux-kvm.org/page/SPICE

3. KVM+Spice, http://coolex.info/blog/153.html

4. 

Keystone Data Model

Data Model

Keystone was designed from the ground up to be amenable to multiple styles of backends and as such many of the methods and data types will happily accept more data than they know what to do with and pass them on to a backend.

There are a few main data types:

• User: has account credentials, is associated with one or more tenants
• Tenant: unit of ownership in openstack, contains one or more users
• Role: a first-class piece of metadata associated with many user-tenant pairs.
• Token: identifying credential associated with a user or user and tenant
• Extras: bucket of key-value metadata associated with a user-tenant pair.
• Rule: describes a set of requirements for performing an action.

While the general data model allows a many-to-many relationship between Users and Tenants and a many-to-one relationship between Extras and User-Tenant pairs, the actual backend implementations take varying levels of advantage of that functionality.

Keystone Service Backends

Service Backends

Each of the services can configured to use a backend to allow Keystone to fit a variety of environments and needs. The backend for each service is defined in the keystone.conf file with the key driver under a group associated with each service.

A general class under each backend named Driver exists to provide an abstract base class for any implementations, identifying the expected service implementations. The drivers for the services are:

• keystone.identity.core.Driver
• keystone.token.core.Driver

If you implement a backend driver for one of the keystone services, you’re expected to subclass from these classes. The default response for the defined apis in these Drivers is to raise a keystone.service.TokenController.

KVS Backend

A simple backend interface meant to be further backended on anything that can support primary key lookups, the most trivial implementation being an in-memory dict.

Supports all features of the general data model.

SQL Backend

A SQL based backend using SQLAlchemy to store data persistently. The keystone-manage command introspects the backends to identify SQL based backends when running “db_sync” to establish or upgrade schema. If the backend driver has a method db_sync(), it will be invoked to sync and/or migrate schema.

PAM Backend

Extra simple backend that uses the current system’s PAM service to authenticate, providing a one-to-one relationship between Users and Tenants with the root User also having the ‘admin’ role.

Templated Backend

Largely designed for a common use case around service catalogs in the Keystone project, a Catalog backend that simply expands pre-configured templates to provide catalog data.

Example paste.deploy config (uses $ instead of % to avoid ConfigParser’s interpolation):

[DEFAULT]
catalog.RegionOne.identity.publicURL = http://localhost:$(public_port)s/v2.0
catalog.RegionOne.identity.adminURL = http://localhost:$(public_port)s/v2.0
catalog.RegionOne.identity.internalURL = http://localhost:$(public_port)s/v2.0
catalog.RegionOne.identity.name = 'Identity Service'

LDAP Backend

The LDAP backend stored Users and Tenents in separate Subtrees. Roles are recorded as entries under the Tenants.

OpenStack Storage Design Mode

Table 5.1. OpenStack Storage

	Ephemeral storage	Block storage	Object storage
Used to…	Run operating system and scratch space	Add additional persistent storage to a virtual machine (VM)	Store data, including VM images
Accessed through…	A file system	A block device that can be partitioned, formatted and mounted (such as, /dev/vdc)	REST API
Accessible from…	Within a VM	Within a VM	Anywhere
Managed by…	OpenStack Compute (Nova)	OpenStack Block Storage (Cinder)	OpenStack Object Storage (Swift)
Persists until…	VM is terminated	Deleted by user	Deleted by user
Sizing determined by…	Administrator configures size settings, known as flavors	Specified by user in initial request	Amount of available physical storage
Example of typical usage…	10 GB first disk, 30GB second disk	1 TB disk	10s of TBs of dataset storage

If you only deploy the OpenStack Compute Service (nova), your users do not have access to any form of persistent storage by default. The disks associated with VMs are "ephemeral", meaning that (from the user's point of view) they effectively disappear when a virtual machine is terminated. You must identify what type of persistent storage you want to support for your users.

Today, OpenStack clouds explicitly support two types of persistent storage: object storage and block storage.

 Object Storage

With object storage, users access binary objects through a REST API. You may be familiar with Amazon S3, which is a well-known example of an object storage system. If your intended users need to archive or manage large datasets, you want to provide them with object storage. In addition, OpenStack can store your virtual machine (VM) images inside of an object storage system, as an alternative to storing the images on a file system.

Nova Networking Design Mode

Type	Strengths	Weaknesses
Flat	Extremely simple. No DHCP broadcasts.	Requires file injection into the instance. Limited to certain distributions of Linux. Difficult to configure and is not recommended.
FlatDHCP	Relatively simple to setup. Standard networking. Works with all operating systems.	Requires its own DHCP broadcast domain.
VlanManager	Each tenant is isolated to their own VLANs.	More complex to set up. Requires its own DHCP broadcast domain. Requires many VLANs to be trunked onto a single port. Standard VLAN number limitation. Switches must support 802.1q VLAN tagging.
FlatDHCP Multi-host HA	Networking failure is isolated to the VMs running on the hypervisor affected. DHCP traffic can be isolated within an individual host. Network traffic is distributed to the compute nodes.	More complex to set up. By default, compute nodes need public IP addresses. Options must be carefully configured for live migration to work with networking.

Quanum Network Deploy Legacy Case

Quantum把網路分成"network"、"subnet"、"port"等三個部分

1. network: 

network為虛擬的switch

2. subnet:

ip區段(including ipv4 、 ipv6)，主要是用來分派位於某個特定Network內虛擬機器的IP位址，而每個Quantum L2 Network可以擁有多個Subnets

3. port:

至於port的部分，則是用來代表在某個Quantum L2 Network的虛擬埠號(Virtual Port)之對應。簡單來說，可以把它視為在某個虛擬交換器當中的虛擬埠(Virtual Port)。

• Single Flat Network

• Multiple Flat Network

          每個tenant可以看到多個共享網路(shared networks)並選擇想要將VMs附加到哪一個共享網路底下

• Mixed Flat and Private Network

• Provider Router and Private Networks

• Per-Tenant Router and Private Networks

quantum metadata agent

quantum-metadata-agent  推測為openstack grizzly的new feature。

The OpenStack Compute service allows VMs to query metadata associated with a VM by making a web request to a special 169.254.169.254 address. Quantum supports proxying those requests to nova-api, even when the requests are made from isolated networks, or from multiple networks that use overlapping IP addresses. Enabling this requires setting the following fields in nova.conf


http://docs.openstack.org/trunk/openstack-network/admin/content/nova_config_metadata.html

Quantum 安全群組（Security Group）設定

edit  /etc/nova/nova.conf  in Compute Node:

firewall_driver = nova.virt.firewall.NoopFirewallDriver
security_group_api = quantum

Data Forwarding Node in Quantum

Data Forwarding Node

需要安裝 openvswitch 以及quantum-plugin-openvswitch-agent agent

All hosts running quantum-plugin-openvswitch-agent also requires that an OVS bridge named "br-int" exists. To create it, run:

ovs-vsctl add-br br-int

CentOS: /etc/sysconfig

Reference:
http://www.centos.org/docs/5/html/Deployment_Guide-en-US/ch-sysconfig.html

Xenserver Plugins on OpenStack

https://github.com/openstack/nova/tree/master/plugins/xenserver/xenapi/etc/xapi.d/plugins

_bittorrent_seeder 

Seed a bittorent image. This file should not be executed directly, rather it should be kicked off by the `bittorent` dom0 plugin.

agent

XenAPI plugin for reading/writing information to xenstore

bandwidth

Fetch Bandwidth data from VIF network devices

bittorrent

Download images via BitTorrent.

glance

Handle the uploading and downloading of images via Glance.

kernel

Handle the manipulation of kernel images.

migration

XenAPI Plugin for transferring data between host nodes

pluginlib_nova.py

Helper functions for the Nova xapi plugins.  In time, this will merge
with the pluginlib.py shipped with xapi, but for now, that file is not
very stable, so it's easiest just to have a copy of all the functions
that we need.

utils.py

Various utilities used by XenServer plugins

workarounds

Handle the uploading and downloading of images via Glance.

xenhost

XenAPI plugin for host operations

xenstore.py

XenAPI plugin for reading/writing information to xenstore

Host Aggregate : 用來在availability zone中進行劃分

Host aggregates are a mechanism to further partition an availability zone; while availability zones are visible to users, host aggregates are only visible to administrators. 

[Note]

Each node can have multiple aggregates, each aggregate can have multiple key-value pairs, and the same key-value pair can be assigned to multiple aggregate.

Xenserver

當使用Xenserver當作hypervisor時，可以使用host-aggregate來管理Xenserver Resource Pool以支援Live migration。