Thu. Jun 17th, 2021
    Subgovernances Efficiency Analysis

    The community held a contest to evaluate the work of Free TON subgovernances — specialized groups engaged in the project development in specific areas: software development, decentralized finance, media, design, analytics, etc.

    The contest results do not allow us to make a clear conclusion which of the subgovernance was the most successful, but they present interesting ideas for the future.

    In 2020, the Free TON community has held many events related to partnerships, specialization of activities within subgovernances and organizing various contests. All this work should contribute to the project development, and especially the gradual implementation of the principle of decentralization — the involvement of various participants and the distribution of tokens between them.

    Being inside such a significant and impetuous process, it is impossible not to wonder about the success of what has been done and the correctness of the chosen directions. To understand the current situation and answers to important development questions, the Analytics&Support Subgovernance came up with a number of initiatives:

    • held a contest to analyze the effectiveness of the jury members responsible for judging the competitions;
    • held a contest to analyze the performance of Free TON subgovernances;
    • planned to evaluate the effectiveness of Free TON partnerships.

    The first two contests started at the same time and some of participants applied for both, which may have reduced the degree of elaboration of their works. At the same time, the subgovernances efficiency analysis contest turned out to be more difficult for the participants. Only the submission that took first place scored more than six points (7.6, to be exact). Among the rest, who got into the top ten, there were submissions with an average score below five.

    The contest was held from November 24 to December 12, 2020. At that time, there were 10 subgovernances in the Free TON community. At the moment there are 16 of them.

    We present to your attention the main results and conclusions of the contest.

    Contest Winners

    First place

    The highest score was received by submission No.25. Its author managed to take first places in two contests at once — in the Subgovernances Efficiency Analysis and in the Jurors Efficiency Analysis. In both works, the competitor uses the same approach: develops an interactive interface model presenting statistical data and the final indicators of the subdivisions. All information is placed on interactive panels (Dashboards) and is grouped into three pages:

    • Subgovernances Motivation

    The main information relates to the use of tokens in subgovernances: the number of requested, received and spent both for contests and for administrative activities. There is a comparison of data from different subdivisions in the form of diagrams.

    • Contests

    The page displays various parameters of the contest activity of subgovernances. On their basis, a number of characteristics are identified. The relationship between them forms the subgovernance performance indicator (KPI). As characteristics are used such as: the number of subgovernances, contests, submissions, winners, judges and reward sizes.

    • Spreadsheets

    On this page, you can create a data table based on the selected parameters for subsequent export.

    A strong feature of the work is the possibility of its further use as a convenient tool for collecting, analyzing and displaying information on the subgovernance activities. The indicators suggested by the author can be supplemented with other options. Therefore, subsequent implementation will be the best justification for this victory.

    Second place

    Work No.13 is much more in line with the formal contest requirements.

    First, it offers three groups of metrics, each characterizing the success of subgovernance in one of the following areas:

    • Media Metrics
    • Organization Metrics
    • Contest & Payout Metrics

    Second, the author collected and analyzed a significant amount of data on 9 subgovernances.

    Third, by means of formulas, the data are converted into percentage indicators characterizing the activities of subgovernances within each of the three areas.

    Fourth, on the basis of percentage indicators, a rating of subgovernance was formed within each group of metrics and for all three groups of metrics.

    As you can see, the use of the comparative method for evaluating efficiency provides a clear picture of the subgovernance ranking.

    Along with a positive assessment of the work, the judges questioned the advisability of performance indicators and transparency of calculations.

    However, before agreeing with the opinion of the jury, it is worth paying attention to this fact:

    The content of many works demonstrates the attempts to solve a specific dilemma: clear indicators do not reflect reality, but indirect indicators do not meet the contest requirements.

    What does this mean?

    On the one hand, the available quantitative parameters of the activities of subgovernances are not enough for the development of appropriate and comprehensive performance indicators. Therefore, the proposed options look far-fetched, detached from reality, arbitrary.

    On the other hand, the authors’ attempt to move away from creating clear criteria in comparative analysis and proposals for the future will inevitably cause criticism in non-fulfillment of requirements and lack of a specific result.

    The author of the described submission leaned towards the first option, and the work lacked a convincing justification for the choice of indicators and calculation methods.

    Third place

    Work No.16 completes the list of submissions with the highest rewards. Its author does not offer universal performance criteria, but focuses on the analysis of one object. This approach allows him to describe in detail the successes and failures of the specific subgovernance.

    The Web&Design Subgovernance is chosen as an example. Their work was evaluated according to the following criteria:

    • Degree of implementation of the goals set during the creation of the subgovernance: quantitative and qualitative indicators are used.
    • Number of ended / active contests. Evaluation of the results of completed competitions, their effectiveness in accordance with the set goals.
    • Analysis of contests (both current and ended) for completeness of criteria for the admission of works and their assessments.

    The work is a detailed report (32 pages) on the selected aspects of the subgovernance. A large number of materials were processed, and the results are presented in the form of tables and diagrams.

    In addition, the author made a number of comments and suggestions to improve the efficiency of the subgovernance, which was approved by some jury members. It is noteworthy that the author’s ideas were quickly applied and were used for a Greeting Card Design Contest held by the Web&Design Subgovernance. For example, the proposal to add a publicity criterion to popularize creative developments appeared in the form of a contest requirement to post works on a social network.

    Along with a positive assessment, the judges’ remarks also mentioned shortcomings: unclear performance indicators, lack of automation of analysis and study of other subgovernances.

    Other Ideas for Attention

    Some participants questioned the possibility of creating universal criteria, since the effectiveness of subgovernances (their ability to perform work and create the required result) depends on the goals and objectives, working methods, resources involved and their impact on the development of the Free TON project. These parameters differ from one subgovernance to another.

    For example, in work No.22, it is suggested that for subgovernances, the main task of which is the creation of software products and procedures, an indicator can be an assessment of the degree/quality of their development and implementation. Whereas for subgovernances aimed at attracting new users to the Free TON infrastructure, there is an increase in traffic to specialized community resources.

    The implementation of the principle of decentralization as one of the important goals of the for subgovernances was not often mentioned in the works of the contestants. Therefore, noteworthy is the work No.28 (4th place in the rating), in which the author used data on the distribution of positions in for subgovernances among their members. The author shows that about two-thirds of all members of for subgovernances hold only one position (jury or initial). At the same time, 23 people are members of more than one for subgovernances (16 — in two, 5 — in three, 1 — in four and 1 — in five). The competitor believes that such data generally speaks in favor of decentralized governance. The only question is the fact that the number of positions of initials (94) is almost one third higher than the number of positions of judges (69).

    Subgovernances Work Analysis or Assessment of Their Efficiency?

    In general, the work of the contestants is a description and analysis of the subgovernances activities. Only a few were able to assess the effectiveness by proposing specific indicators, which, however, were also criticized. This result is a consequence of the nontriviality of the task.

    Firstly, in order to develop a reasonable and realistic performance indicator, it is necessary to understand both the development strategy of the Free TON project as a whole and the tasks of specific subgovernances.

    It is important to adequately assess the current stage: not to diminish the possibilities, but also not to raise the bar too high. It is not surprising that some of the contestants devoted part of their work to interpreting the concept of efficiency in relation to the activities of subgovernances. This may well be the subject of a separate contest.

    Secondly, before starting to evaluate the effectiveness, it was necessary to collect and process a large amount of data (the contest lasted 18 days).

    It should be taken into account that the primary data of the community are not presented in a form ready for analysis. In addition, quantitative data on some aspects of the activities are not available as such. Contestants had to collect indirect information and, using qualitative methods of analysis, transform it into indicators.

    Thus, the participants tried to answer two global questions about Free TON subgovernances:

    1. What’s going on there?
    2. And how effective is it?

    The answer to at least the first question is already a good result. Never before has the community conducted such a large-scale and comprehensive study of its own activities. So the results of the research of the contestants, their ideas and proposals can contribute to the further development of analytical and predictive activities within the Free TON project.

    There is reason to expect that the Analytics&Support Subgovernance will benefit from the competition and implement the most valuable and practical ideas. We will monitor the further progress of the issue of efficiency in the community.