A System For Self-Organization & Self-Management In Distributed Systems

A generic framework is presented in this paper which supports self-organization and self-management in a hierarchical system. This system allows convenient use of not only self-management but self-organization tactics at each level in the hierarchy.

Consequently, the tactics are eyed as parameterized functions. Information derived from various levels in the hierarchy can be used as parameters. Information which is obtained from below constitutes the status of the lower levels, while information obtained from above can be used to influence not only the direction but also the rate of system evolution. An innovative approach is used to measure the distance between components, where a suitability index has been defined. This type of framework can be used in all hierarchical system, however in this paper explanation is provided for large systems which are structured in a hierarchal manner. The usefulness of this framework can be seen as part of the cloud Lightning project which has been explained in detail below.

This paper explains about a framework which can be used for not only hosting but also implementing the self-organizing and self-managing tactics which are related to different layers in a hierarchical architecture. It relates to how the outcome of these tactics are exchanged and understood by the environments close by and how the purpose of these tactics has an impact on the feedback.

The framework is made up of customized functions, where the system design includes specific requirements and objectives. It also makes use of an operation; this operation contains information which is obtained from a process called directed evolution. Furthermore, the paper also presents various example characteristic functions of design objectives like efficient power management, exploitation of heterogeneous hardware and improved service delivery.

Discussion

The hierarchical architecture shortens the search space which is essential to obtain components which are present on the lower level of the hierarchy. The search space must be shortened when the components have very big quantities and definite properties which requires access to specific components. In this hierarchical system the lower level consists of functional resources, for example, consider a cloud hierarchy which is made up of computer resources. The components which are present in the intermediate level give access to resources.

All the components are made up of different strategies, these strategies explain how the components in the hierarchy which are present at different levels must evolve in order to attain a state which is ideal, this is called as the component’s local goal and the strategies used for this is either self-organizing or self-managing. Primarily, decisions which are made by components in any specific level in the hierarchy can have a direct influence on the adjoining levels. These types of influences can come either from the top levels to the bottom levels or vice versa. When they come from the top levels in the hierarchy the process is called as Directed Evolution. In directed evolution the top most levels have the components, in the levels below, any change in operation or configuration will have to adjust with the goals of the upper layer. Similarly, the levels which are at the bottom in the hierarchy has a direct influence on the levels which are above it, this is called as feedback. The feedback is in the form of tuples called as metrics, they are operation of components which take place at the lower level and they give the levels about it a sense of change or evolution. Perceptions are helpful in being certain about the consecutive Directed Evolution decision. The framework in general Consists of these terminologies as below:

1. Autonomous components: Components maintain a local state and they perform local strategies so that it can reach its goal state.
2. Component State: They are a set of metrics that correspond to the components.
3. A goal state is the state of the component which is highly desired, here all the metrices have ideal values. It is a desired state of the component and the metrics consist of desired or ideal values.
4. Metric: It is measured as the state of the component.
5. Self-organization strategies:These are actions which are performed by various components, these components reside on the same level and this results in change in the components at that specific level.
6. Self-management strategies: The actions performed by the individual components leads them to the goal state.
7. Assessment functions: correspond to the functional components in the lower levels of the hierarchy.
8. Perception: This takes place in the bottom levels in the hierarchy and it is a function of the metric.
9. Impetus: It is acquired from the top levels in the hierarchy and it is a function of weight (which is a factor which corresponds to directed evolution).

Steering is a process which alters the weights and communicates them across the hierarchy. The framework also explains about other processes: The first process communicates impetus; the weights are transmitted from one specific level to the levels which are below it.

The second involves a process which lets the components to communicate the feedback which by transmitting the metric, the transmission of metric takes place to the components which are in the following levels.

The third mechanism involves the components behavior to be modified in response to the impetus and feedback.

The fourth mechanism involves components to self-manage in correspondence to the different strategies that maximizes their individual suitability indices.

The fifth mechanism explains how the collective as well as the individual suitability indices increase because the components in the same level self-organize according to different strategies.

Strategies for Self-organization and Self-management

Strategies are nothing, but parameterized functions corresponding to various levels in the hierarchical system. These levels are partitioned and managed by those levels which are directly above level-1. The levels which are higher that is level 2 to n-1, the decisions are made by them for directing the tasks which enter at level n, and where these tasks must be led for execution. An assumption is made that the entities which are present in every level in the hierarchy can collaborate to attain a common goal. The different types of functions are suitability index, perception, impetus and assessment functions. Assessment function: There are specific objectives which have been proposed which include the following: maximize task throughput, to maximize energy efficiency, to maximize resource management efficiency, to maximize computational efficiency. The objectives for these are expressed in four equivalent assessment functions.

F1: It calculates the throughput for the number of servers presently being used.

F2: It calculates the energy consumption corresponding to the hardware type and its use.

F3: Computational efficiency can be used to calculate using this function, in correspondence to the size of the underlying resources.

F4: It is a function which calculates the management efficiency, corresponding to management costs for resources. Formulas have been derived for these functions, the assessment function can not only be reformed but also redefined according to specialized choices made.

Perception function: Perception is a function of the metrices, it is obtained from a lower level in the hierarchy. Based on the specific assessment function and the specialized choices which were made above, the systems state at any time can be specified as a point in a four-dimensional space.

Impetus function: Impetus is nothing but a function of weights which is attained from the top levels. The weights which are closely related to the impetus are obtained, these are weights in the previous state and the weights which are propagated from the top levels. The constants are chosen randomly to bound the minimum and the maximum value of weights. The impetus is chosen to be the average of weights which are got from the top levels with the existing weights of the component. The averaging function constricts the influence of the upper levels to that of the lower levels so that the system can go through a smooth transition towards a global goal. Suppose we consider the weights which are obtained from a level l+1 at time t+1 and the current weight w at level l then a new weight is obtained which is given by an equation.

Suitability Index: In each of these levels the tasks are led to component with the maximum suitability index, a tie between two components can be avoided where a random value should be added to all the suitability indices before a pathway being selected which is composed of maximum SI values. As a significance of the four functions which were defined above an optimal global metric of a system is achieved without achieving them in every component. There are some separate local goals of the system in correspondence to suitability index in the higher levels because the components have an overall view of the underlying subsystems in terms of an average value. Therefore, ever component tries to optimize its suitability index so that it can obtain more tasks which after evolution will lead to a vector of metrices. In level 1 it is not possible to achieve the metrics, because they are connected to the requested number of resources very strongly. Therefore, the local goal of the partition manager must be to change in a direction which is pointed to by the optimal metric vector with reference to the weights which is stated through the suitability Index.

Simulation and Evaluation with reference to the cloud lightning system. This framework which is defined above is called as SOSM framework in short, it has been simulated and evaluated with respect to a cloud lightning system also called as the CL architecture. With the current cloud deployments in place the Cloud lightning architecture is hierarchically organized. In distinction to these systems, it recommends an innovative and distributed approach in which the components are self-organized and self-managed to deliver a suitable resource solution from a pool of heterogeneous resources as a response to user-initiated service request.

The heterogenous resource pool is thought of as a Cell in the cloud lightning architecture, which is further partitioned into v racks. A cell manager is related to the cell, which will behave as an entry point to the heterogenous resource pool. A v rack manager also called as vRM in short is connected to each v rack, which handle all the resources that can be exposed from its related v rack.

Heterogeneity causes the vRMs to be distinguished by the resource hardware type and the related software stack, which is called as the CL resource type. The children of the cell manager are called p routers and there is one p Router for every one of the distinct CL-Resource type. The p Routers have children which is called the p Switch which partition the vRMs, this leads to manage the same CL resource types into groups. The amount of p switches per p Routers is not fixed over time, nor is the size of the vRM groups managed by each of the p Switch. Therefore, the cloud lightning hierarchical architecture consists of four levels which are CM, pRouter, pSwitch and vRM.

Initial Conditions

The implementation of the simulator takes place in an octave environment. Every resource type is made of three attributes which include

1. Computational capability, this is the performance improvement of the resource in comparison to the performance improvement of the baseline resource type.
2. Power consumption includes the power consumption of the resource in comparison to the power consumption of the baseline resource type.
3. The quantity of servers of this type for each cell. The resources which are of type 1 is used as a baseline. The features of the resource types are as follows.

CompCap refers to the computational capability, PowCons refers to the power consumption and Num Of Servers this is the number of server’s equivalent to each hardware type. The servers can be either of these two states that is consuming maximum power or ideal where the consumption of power is 10% of the maximum.

Every task is made up of a set of parameters such as the time span, which represents the time required to execute the task, resource requirement includes the minimum number/ maximum number of servers needed to complete a task, Available implementations is a vector which is made up of the resources for which an implementation of a task exists. Three types of tasks are taken into consideration.

The process queue, it must be initialized in every iteration which consists of a series of tasks which arrives at the cell at the timestep t. Therefore, the minimum and the maximum number of tasks are given by Nc ∈ [Nmin, Nmax], where Nc is called the number of tasks at any given time. The values which belong to a task are calculated using a random number generator. The process of Directed Evolution is triggered only when the system is significantly loaded. The experiments are all completed with the exact same queue of resource prescriptions so that the result can be unbiased and reproducible. The v Racks original size is set to 50 servers. The number of CM’s for the present trial is set to 1.

Static management strategy: Prescribed values are assigned for the weight vectors of all the components in the static management strategy for the experiments duration, therefore the system reflects on only those parameters which are marked with non-zero values in the vector as significant. An example has been presented where all the metrics are equally important that is they must retain balance. In the figure 3 which is shown below. The system is utilized with respect to time, in the figure it can be seen that the systems use oscillates close to 90%. This is because self-organization has been performed since the system tries to optimize the management costs which are linked directly to the size of the v Racks, that is large v Racks will have a suitability index which has been reduced, this leads to reduced amount of incoming resource prescriptions which will result in the exchange of some or all of its servers to the other v Racks.

Findings

In this Paper an innovative, generic framework has been proposed which supports self-organization and self-management in a hierarchical system. The framework lets the incorporation of local self-organizing and self-managing strategies at every level in the hierarchy and the effects of these strategies can not only be communicated to but also used by the strategies in the adjoining levels of the hierarchy.

Conclusion

Local strategies can be used to accomplish local goals and a global goal state is reached by parameterizing the strategies along with the weights which propagate from the top levels in the hierarchy in a process which is called as directed evolution.

The idea of suitability index was formulated as a function of metrics formulating from the hardware telemetry and the weights which propagated as part of the directed evolution. This measure specifies the distance that a component is from an idealized state of evolution which is essential to maximally add to the global goal. Therefore, the local strategy of each of these components is to maximize its suitability index. This framework has not only been evaluated but it has been implemented with respect to the H2020 Cloud Lightning project, the simulations show that the outcomes have been promising in terms of improved resource utilization and service quality. The tasks which propagate through the cloud lightning system alters the SI of each component on the path taken by the task. Therefore, as the system converges, that convergence intimately relates to the task profiles that were processed.