Proposal: Create An Ecosystem Subgroup Under project-emco

Problem Statement

In the current organization of project-emco, we have 2 subgroups: core and ui. Underneath these are specific git repositories, such as emco-base and emco-gui. This organization has served well so far, but there are gaps which are exacerbated by newer developments. Here are some of those gaps:

  • With the ongoing work to integrate Temporal workflows into EMCO, we need a place to put example/reference workflows.
    • The workflows are logically independent of core EMCO, so placing workflow documentation and code snippets inside the core/emco-base is not appropriate.
    • The workflows need to be compiled/built and deployed. The deployment may be via EMCO. IOW, a workflow is similar to an app in this context. This means each workflow needs its own build environment and deployment environment. This means it requires a repo of its own. (Intel is seeding this work with a reference workflow, and folks want it soon, so this has become rather urgent.)
    • A workflow may be useful to many organizations, so they may develop it collectively. This also necessitates the need for a separate repo for each workflow.
    • Apart from the need for a build environment, we want to enable developers to contribute back.
  • In the future, we may have 3rd party EMCO controllers, which are not part of the core/emco-base: that is, they would not be built and released with the core, but would be collectively developed and used in the open. Similar to workflows, this scenario also requires separate repos outside the core.
  • It would be great to have a 'Best of EMCO' kind of repo, similar to 'best-of' lists and 'awesome' lists in github. This could be a collection of EMCO resources, blogs, use cases etc. This also would not fit either under core or under ui

Solution (updated based on comments below):

Let us create a new subgroup under project-emco to keep non-core contributions from the EMCO ecosystem, whether they be workflows, 3rd party controllers or non-software resources. We could be wildly imaginative and place it under https://gitlab.com/project-emco/ecosystem.

Secondly, to serve the immediate need for housing Temporal workflows, let us create a second-level subgroup under ecosystem named temporal, i.e., at https://gitlab.com/project-emco/ecosystem/temporal. Under the temporal subgroup would be a git repository that houses the specific reference workflow that Intel is providing. Since that reference workflow's purpose is to migrate a composite application from one cluster to another, that repo would be called migrate-workflow. Putting it all together, the location of the repo would be https://gitlab.com/project-emco/ecosystem/temporal/migrate-workflow.

This proposal allows for more Temporal workflows to be added under the temporal subgroup. Outside the temporal subgroup, we could add workflows for other engines, 3rd party controllers, non-software resources such as awesome lists, and so on. The proposal intentionally does not specify those future possibilities: they can be taken up as the need arises. 

The ecosystem subgroup is proposed as a flat space under which the community-contributed repositories reside. For example, the reference workflow that Intel plans to provide would be under https://gitlab.com/project-emco/ecosystem/temporal-migrate-workflow.

There is no expected need for a PTL for the ecosystem subgroup at this point. A PTL usually sets roadmaps for releases, executes releases and oversees common architecture/API/coding standards. The ecosystem repos may not have regular releases and, even if they do, are not expected to be part of the emco-base releases. That is why we don't see the need for a community PTL.



  • No labels

5 Comments

  1. Overall I agree with a community/ecosystem like subgroup. This is something I also proposed in an early TSC meeting but for KISS reasons we didn't adopt it back then.

    But let me first understand more about the context of the problem statement:

    You mention a reference workflow that relates to Temporal. My question is.. does it relate to EMCO? (outside of the Temporal workflow feature we are doing in EMCO). Reading your statement sounds like it doesn't actually relate to EMCO.

    1. The reference workflow is like a 3rd party app that can be deployed by EMCO. This specific workflow also calls EMCO APIs, though not every workflow is required to do that. That is why it is logically not a part of the emco-base repo but belongs in the ecosystem.

      The name of the 3rd level (temporal) is up for debate. We could choose to have a neutral name like workflows, and have workflows for different engines (temporal, camunda, etc.) coexist there.

      If we start with a two-level structure (just project-emco/ecosystem with all repos underneath that), migrating to a three-level structure will break users because the Go module name reflects the location: going from project-emco/ecosystem/temporal-migrate-workflow to project-emco/ecosystem/temporal/migrate-workflow  will break imports. We face this even in EMCO's dependencies where the URL does not match the module path and name. So, it will be good to decide on a clean structure right at the beginning. 

      1. Yes, right from beginning is certainly better.

        EMCO is a small project, at least right now, and yet it already has repositories arranged within subgroups (core, ui). Compare that to the much larger K8s where they have an official flat structure for all their repos (inside top-level kubernetes GitHub group).

        Deep hierarchies sound good on paper (structuring everything in the right box). Heck, I'm guilty of doing it too much myself.

        But it's impossible to achieve perfection when it comes to this. Example:

        day 1: let's create project-emco/ecosystem/temporal/migrate-workflow

        day 2: let's create the same kind of thing but for the Camunda engine this time... hmm.. project-emco/ecosystem/camunda/migrate-workflow?

        (extra.. the repository name is actually the same, so if you try to clone them both without locally renaming, they will clash)

        day N: oh.. looks like we have 20 subsubgroups with a total of 20 repositories (1 in each). Messy.


        If the objective is to have things organized tidily, I vote strongly for a 2-level, rather than 3-level approach.. let's use the names of the repositories (first and foremost) to express what the repositories refer to (that's also why we chose /core/emco-base rather than /core/base or /core/emco).


        I like having an ecosystem/community/incubation subgroup. But I question whether it would be better to stick to conceptual categories instead.

        I.e., ecosystem-provided projects could live in any subgroup.

        Let me illustrate - should we do:

        /core/

        /ui/

        /workflow/

        (and others)


        And allow community-contributed repos in any of these subgroups? (with TSC approval)

        (examples: /ui/yourcompany-ui, and /workflow/temporal-migrate-workflows)

        1. I see your concerns. If we start with 2 levels, decide on a 3-level structure in the future, and do not relocate existing repos (i.e. only new repos go into the 3rd level), then there is no breakage. Or, any relocation of an existing repo needs to be discussed in the open community and must get alignment from the main committers – so that breakage can be foreseen and handled. If we can stipulate that, we can go with 2 levels even now. 

          The suggestion that "ecosystem-provided projects could live in any subgroup" can also result in issues. Say we have a workflows/ subgroup and we put migrate-workflow repo there. We could have an intent-examples/ subgroup tomorrow, but the migrate-workflow is also an example of workflow intents. In general, there can be cross-cutting properties across subgroups.

          The other problem is that, the community-driven repos will (probably) not be built and released along with the core emco. Furthermore, there is no need for architectural/design/API overview of these community repos. So, there is no need for a PTL for the community repos. If those repos get interspersed with core/, ui/, etc., that may complicate the life for the core and ui PTLs. 

          So, in summary, we could keep part 1 of the proposal (https://gitlab.com/project-emco/ecosystem) and modify part 2 to place the migrate-workflow right under the ecosystem subgroup (subject to the first paragraph above in my response). Is everybody ok with that?

          1. I'm good with this and I agree with the intents vs. workflows example.