Seeking Your Input and Expectations on the Libraries and Frameworks Working Group

[yordis]

As you may know, we recently established the Libraries and Frameworks Working Group to ensure that BEAM languages have a rich, vibrant ecosystem and a high degree of developer experience.

We aim to provide resources for library and framework authors, promote best practices on library and framework standardization, and improve the visibility and usability of our library ecosystem.

This effort will only be possible with your active participation and invaluable feedback. So, we would like to invite you to share your understanding, expectations, and what you want to see from this working group.

This discussion seeks to achieve two critical objectives:

• Gather Feedback on Your Understanding of the Group: Your thoughts will help us determine if the group's mission and objectives are communicated and understood. That includes our short-term and long-term deliverables, our commitment to the community, and why we believe the Foundation's involvement is crucial in achieving our goals. We hope to refine our messaging based on your feedback, ensuring that our mission resonates with all.
• Understand Your Expectations and Interests: What would you like to see us achieve? Would you like us to address any specific challenges you face as a developer or library author? Are there any areas of interest we should focus on? Your insights will guide our strategic planning and help us prioritize the most beneficial initiatives for the community.

We value your opinions and would be grateful if you could take a few moments to share your thoughts. Your feedback is essential to ensure the Libraries and Frameworks Working Group meets the needs of the BEAM language ecosystem and serves the community in the best possible way.

[zachdaniel]

Thanks for getting this all started. Since I was a part of the initial writing up of the working group, I'll leave it to others to provide their thoughts. One important thing to remember is that these things will take some time, and since this group is working from all over the world, we'll want to leave ample time in discussions for people to respond and provide their thoughts. Slow and steady wins the race :)

With that said, putting in the effort to keep us all on track is necessary and you're doing a great job 🙇

[yordis]

Slow and steady; Slow and steady! Let's pause for at least 4 weeks to revisit this. I am gonna be observing but I am gonna avoid replying at all to let their opinions drive the conversation first!

I am gonna share eventually my thoughts.

[zachdaniel]

I'd say we only need to give everyone a week, maybe two to get their thoughts together, 4 weeks might be overkill 😆

[Theosaurus-Rex]

Thanks for starting this! Busy Australian here so will be as involved as work/life/timezones permit 😅

To summarise my understanding of the working group: Its primary purpose is to support library and framework authors within the BEAM ecosystem to ensure that both authors and the developers using and contributing to their work have a robust, supportive, and pleasant experience.

A bit about my perspective - I'm currently working as a full-stack consultant who primarily works with Elixir and Phoenix LiveView, as well as the Ash Framework, in my day-to-day code. As an early adopter of Ash who picked it up while I was still learning the ropes in Elixir, I've come to understand just how important it is that authors are able to clearly communicate and document their work to ensure a smooth onboarding experience for developers. In addition to easing the learning curve for devs picking up these tools for the first time, having a robust culture of documentation eases the burden on authors in terms of being available to answer user questions, allowing them more time to continue adding to and improving their libraries and frameworks.

As such, my primary interest for volunteering, and where I feel I am most useful as a relatively new developer, is around finding the best methods and strategies for effective documentation, as well as ongoing maintenance of existing library and framework documentation.

[woutdp]

Thanks for setting this up, excited to see where this leads to!

I'm the developer of live_svelte and it was my first foray into developing an Elixir library. I hope to provide some useful input and ideas for the group in the coming future. Mostly have input on the Elixir front but I understand the group envelopes all BEAM languages.

What I understand from the group, and it's purpose, is to improve the developing experience for library authors in the BEAM ecosystem, as well as the consuming experience for developers using libraries. The group's goals are to improve this experience, and they will do so by discussion, suggestion and standardization.

I'm particularly interested in documentation, and making beginners (to programming in general or to BEAM) feel welcome to Elixir and the BEAM by having a great ecosystem to learn from and explore in. One thing I'd love to see is a 'standard template' to start libraries from, with a documentation setup that feels clean and well thought out. Another thing I'd want to do is lead by example, by conforming all our own libraries to a certain set of standards that new library authors can learn from.

[barnabasJ]

I was especially reminded of this https://hexdocs.pm/elixir/main/library-guidelines.html when I heard about this WorkingGroup first.

[zachdaniel]

Agreed, I think one of our main tasks will be reviewing that, and keeping it up to date.

[Nezteb]

I agree with everything that others have said.

To add another idea, I think it's important to promote library/framework discoverability and make it easy for developers to find what they might need. In particular I often look at:

• https://github.com/h4cc/awesome-elixir
• https://www.libhunt.com/l/elixir

We could also provide some guides for library authors about how they can fund their work via platforms like GitHub Sponsors, Patreon, OpenCollective, LiberaPay, etc.

[zachdaniel]

I wonder if this can be added as a feature directly to hex. Like if library authors can configure categories, and anything w/ a certain number of downloads or GitHub stars in a given category is shown on a “best of” page or something like that.

[zachdaniel]

I also think this group may end up contributing to hex and ex_doc, that kind of thing.

[yordis]

I once proposed to the Hex folks to extend the website to include even more metadata, but it was rejected. They want to focus (back then) just on the package management aspect only.

Being said that was a long time ago.

After gathering the feedback, I will focus on solution space to see the overlap in ideas and whatnot. But I love this way too much 😄

[TwistingTwists]

+1

Adding to above, a semantic search for libraries would be good for discoverability. For example, newcomers to the ecosystem often search like

"Library to upload to s3"
"Best library for authentication and authorisation"

I recognise that this particular "semantic search" topic may be way too experimental at the moment for consideration.

So, maybe just a newsletter of good blogs / docs / codebases focussed on exploring the elixir ecosystem - every 3-4 months. (or xyz frequency)

[tiagoefmoraes]

I agree with what others said and think having best practices around CI for hex packages would be nice. A resource that comes to mind is https://github.com/felt/ultimate-elixir-ci.

[TwistingTwists]

I agree with ideas above. One aspect that I would like to highlight is - demo-able examples of ecosystems that have a lot of libraries.

Example

1. membrane framework. It has more than 10 things that need to work together. And that is hard to get right unless, it is in a demo repository/ docs. Lot of experienced developers have been asking for help on this aspect on their discord.

2. Ash. Most of ash libraries work with each other at the moment. At some point in future, it would be useful to have a way to document "what versions of one library work with the other".

[yordis]

As I discussed with some of you, here is a list of things I would like to see in the ecosystem:

• Create a website to enhance Hex with more metadata that helps the ecosystem.
• Extent and improve the quality of packages by providing more guidelines and tooling.
• Align the ecosystem regarding packages, and create a more decentralized but aligned ecosystem striving for a platform at a higher layer than the language layer provides today. Like a standard library that isn't part of the language layer per se. A place where we could experiment and evolve things into the language layer.
• Gather information about what packages, improvements, and whatnot could be done across the ecosystem.
• Find a sustainable way to support OSS development.
• Have a place where the package ecosystem has a community where we support each other as a huge team.

My primary goal is to create a more aligned ecosystem.

---

My most immediate task: create a proposal (ideally at the Erlang level probably) to introduce Feature Flagging (like Rust, as an example) that would allow Language Layer but also package maintainers or even Elixir/Erlang peeps to test features without committing ahead of time to it. App devs to tree-shake their apps without asking OSS maintainers to take the burden of creating tiny packages (packages like Ueberauth, Tesla, Ash (I believe), Swoosh, and many others that have all the adapters built-in, but we barely use all of them).

From a simple point of view: Hex packages are for the distribution of an artifact, while Feature Flagging helps you know exactly what you would like to have compiled in your application.

Have a robust way to deal with existing pain points between App and Package devs.

---

• Language Layer: I mean Elixir, Erlang, [insert any language here], BEAM

[Neophen]

For me the most important thing missing in the ecosystem, is courses/blogs/documentation.
While hex documentation is good. It's lacking compared to something like Laravel/Vue.

• Improve hex documentation to have nicer templates/layout if that is possible.
• Teach/Help people write extensive documentation with examples/guides/tutorials.

Everything else the previous people covered.
:D

[zachdaniel]

Alright! We've got tons of great ideas, and I think a good shared understanding of what we're trying to improve in general. I think a common theme is that there are "big" ways that we can help, in addition to "small" ways that we can help (in terms of time investment/requiring programming).

Big:

• implementing feature flagging in hex packages
• solve for multi-package documentation & experience on hex docs
• creating a site that wraps hex with additional metadata & features
• provide documentation templates & guidelines for writing good library & framework documentation.

Small:

• built in documentation review
• contributing documentation in hotspots that are under-documented
• finding, curating & elevating visibility of packages, blog posts, articles and things like this. (i.e this could just be adding new packages to awesome-elixir).

Maybe big or small:

• CI tooling to simplify package maintenance and release.

Let me know what can also be added to this list. Once we compile this list, I think our best way to start making progress is to choose one big thing and start on it as a group. I also know that, for example, @yordis is doing some work towards creating a new site that wraps hex, and may perhaps like to make progress on that concurrently with whatever we do. Thats cool with me :) Then, we can also start making a list of small things we can do, and try to put in a little bit of work each week towards those things. How does that sound?

[TwistingTwists]

❤️ 💛

[yordis]

@zachdaniel I would like to use all my energy on the feature flagging.

While also continue gathering information about the enhanced metadata website.

As a follow-up, I am compiling all the responses and will probably use ChatGPT to analyze the sentiment and create a word cloud.

And lately, proposed some stuff I have done a long time ago and CI that definitely simplifies the package maintenance and it will help drastically the existing ecosystem, especially for newbies.

[zachdaniel]

👍 sounds good! I'll follow up here in the next few days. Anyone who has things to add to the list I put up, make a comment and we'll edit the list accordingly too :)

[TwistingTwists]

proposed some stuff I have done a long time ago and CI that definitely simplifies the package maintenance and it will help drastically the existing ecosystem, especially for newbies.

Please add the last bit of comment by Yordis to the list.
It may be added as a part of small / big thing about CI/CD stuff.
:)

[Neophen]

Sounds like an amazing plan. 🚀

[yordis]

Greetings everyone,

I'd like to introduce another perspective on data visualization. However, kindly bear in mind that the information is generated by an AI bot, so its usefulness may vary based on your specific needs.

We will now be closing this particular conversation. Nevertheless, your feedback is crucial to us. We encourage you to keep sharing your insights and suggestions via our Discussion section!

---

Expectations:

1. Adequate time for responses and discussions due to participants' different time zones (mentioned by zachdaniel).
2. Implementation of strategies for effective documentation and ongoing maintenance of existing library and framework documentation (mentioned by Theosaurus-Rex).
3. A 'standard template' to start libraries from, with a clean and well-thought-out documentation setup (mentioned by woutdp).
4. Enhanced library/framework discoverability for developers (mentioned by Nezteb).
5. Guides for library authors about how they can fund their work via platforms like GitHub Sponsors, Patreon, OpenCollective, LiberaPay, etc. (mentioned by Nezteb).
6. Establishing best practices for Continuous Integration (CI) specific to hex packages (mentioned by tiagoefmoraes).
7. Demo-able examples of ecosystems that have a lot of libraries, including documentation on "what versions of one library work with the other" (mentioned by TwistingTwists).
8. Creation of a website to enhance Hex with more metadata (mentioned by yordis).
9. More guidelines and tooling for extending and improving the quality of packages (mentioned by yordis).
10. Gathering information about what packages, improvements could be done across the ecosystem (mentioned by yordis).
11. A place where the package ecosystem has a community to support each other (mentioned by yordis).
12. Improvement in hex documentation, including nicer templates/layout (mentioned by Neophen).
13. Assistance to people for writing extensive documentation with examples/guides/tutorials (mentioned by Neophen).

Interests:

1. Theosaurus-Rex: Interests revolve around finding the best methods and strategies for effective documentation.
2. Woutdp: Interested in documentation, and making beginners feel welcome to Elixir and the BEAM by having a great ecosystem to learn from and explore in.
3. Nezteb: Interested in promoting library/framework discoverability and funding for library authors.
4. Tiagoefmoraes: Interested in best practices around CI for hex packages.
5. TwistingTwists: Highlighted the need for demo-able examples of ecosystems that have many libraries.
6. Yordis: Interested in creating a more aligned ecosystem, promoting feature flagging, and addressing existing pain points between App and Package developers.
7. Neophen: Interested in improving documentation throughout the ecosystem.

---

Documentation (20 mentions): This is the most frequently mentioned term, highlighting the group's primary interest in improving documentation. This includes creating standards, templates, and strategies for effective documentation, encouraging extensive writing, and maintaining documentation for library and framework authors.

Ecosystem (12 mentions): This indicates the participants' focus on enhancing the overall BEAM ecosystem. Expectations around this involve improving the developer experience, package discoverability, quality of packages, and fostering a supportive community.

Packages (9 mentions): The participants discuss the need to improve the developing and consuming experiences for library authors and developers using libraries, implying that there is interest in enhancing the quality and utility of packages in the BEAM ecosystem.

Library/Libraries (8 mentions): A key focus in the discussion, it refers to the need to support library authors better and improve the developers' experience using and contributing to their work.

Hex (7 mentions): This shows the participants' interest in improving Hex, a package manager for the Erlang ecosystem, with enhancements like feature flagging, better documentation, and a website with additional metadata.

Experience (6 mentions): This reflects the goal to improve the experience for both library authors and developers using and contributing to their work. It can be actioned by enhancing the onboarding experience, supporting library authors, and refining the BEAM ecosystem.

Guidelines (5 mentions): This suggests that the participants want clear guidelines for tasks like writing effective documentation, extending and improving package quality, and best practices for CI.

Templates (4 mentions): Reflects the expectation to have standard templates for starting libraries and improving Hex documentation.

Standards (4 mentions): Indicates a desire to establish standards across the BEAM ecosystem, potentially for documentation, package development, and library creation.

Community (3 mentions): Emphasizes the expectation for a supportive community for the package ecosystem.

---

Using these data points, you could prioritize the expectations identified earlier as follows:

Improve Documentation: This task seems to have high urgency, importance, and stakeholder interest, and might not require much resources or time to get started. It also has low risks involved.

Enhance the BEAM Ecosystem: This task is very important for the long-term success of the project, but it could be a high-effort task with many dependencies.

Create Standards and Guidelines: This task is crucial for maintaining the quality of the project, but it might require a considerable amount of resources and time.

Improve Hex: This is important, but since it might require a lot of effort and is dependent on other tasks (like improving documentation), it could be lower on the priority list.

Supportive Community: While fostering a supportive community is important, it is a long-term, ongoing effort that does not have a definite end. It can be done in parallel with other tasks.

---

[zachdaniel]

Great breakdown. Time to make some breakdowns and get to work! Perhaps you could open some additional discussions for more specific initiatives?

[yordis]

Absolutely! That will be the next step. I think that, at the very least, the Feature Flagging and CI topics could be something we could open the door to.

[D4no0]

Maybe I missed it, sorry if I did, however one feature I would love to see is possibility to hand the library to organization.

Let's say I create SMTP library in elixir, and because of lack of time/interest or the fact that I want to state that the library is up to your standards, I would like to be able to have a process of handing the library to your organization, you would review the code and if it is up to your standards, you would accept it and mark the group as the maintainer. This solves problem of dead/poor quality libraries by saying 2 things:

1. Library is in maintenance of group, this means that it is up to standards;
2. If you want to improve it, you can address to the group, there is a much better chance, because usually this is a problem, and yes I know you can fork the library and do your changes, however the problem is still present when we talk about hex publishing, as the author has the sole access.

[zachdaniel]

This is a very interesting concept. I think it could make sense to support "signing over" packages to this group/the foundation, but I'm not sure that it makes sense for the WG to be vetting improvements/features. That is less about responsibility and more about the effort that goes into understanding and maintaining a library. I think the idea is that we would hold on to packages that are critical, and we would coordinate:

• critical bug fixes with no reasonable workaround
• security issues
• language version compatibility

The idea would be that someone else can take over maintenance in the future and in that case improvements and features can be added.

Thoughts?

[D4no0]

I think that what such organization accomplishes in the first place is the concept of "trust", or correct me if I'm wrong. You could manage a list that would classify external contributors such as: neutral, trusted, blacklisted.

A simple example is: I Daniel, a neutral contributor come and contribute to some of your managed libraries, we make some great changes and they are all checked and merged, I increase my trust in the context of my name in your organization, without necessarily being a member of the organization.

Once I contribute a good number of times to the libraries I become a trusted member, this way I can have the "maintainer" status over some libraries (let's say the ones I contributed to), not maintainer in the direct sense, but in the sense that you will approve my MRs without having to resort to some kind of strict checks, and take in considerations MRs that I approve for example.

You have the full authority over the library, however trusted members can have partially more control, without the need to involve maintainers or other people into this game. This should in theory solve the problem of finding people capable of maintaining each library inside of the organization.

[zachdaniel]

I'm not convinced this is reasonable from a time investment perspective, or would yield tangible results in the long run. A lot of these effects already happen organically, or are "outsourced" to individual libraries. For example, the people on Elixir core team, Phoenix core team, or have made PRs to those things are essentially carrying around their credentials.

If we are a mechanism of "trust" then we'd need to have some very rigorous and rock solid procedures for giving, maintaining, and revoking that trust, IMO. I'd also worry about the liability (ethical, not legal) of us saying "you can trust this person or package" and them doing something malicious. I think there are other more attainable things that we can do as a group in the short term.

[D4no0]

Well I guess you are right in terms of time and resources spent, and you are right about other corporations like dashbit that release packages under their namespace and maintain them. I would still love to see a way to allow somehow passing of maintainer status to libraries in a official way, without the primitive things like forks and renaming of libraries, because they are bad from discoverability point of view.

I will investigate this more in terms of other communities and maybe come back with a better variation of the proposal.

[max-au]

Similar requests have been denied in the past. It may create unfair advantage for selected libraries, and also presents a liability that the EEF cannot support.

However it is possible to leverage the EEF to find engineers who think alike! And therefore, willing to share the responsibility of maintaining a library important for the community.

[benoitc]

Hi, I'm intrigued. Why would the foundation release some libraries or packages? How would it keep it neutral against the ecosystem? Benoît Le jeu. 29 juin 2023 à 17:51, Yordis Prieto ***@***.***> a écrit :

As you may know, we recently established the Libraries and Frameworks Working Group <https://erlef.org/wg/libs-and-frameworks> to ensure that BEAM languages have a rich, vibrant ecosystem and a high degree of developer experience. We aim to provide resources for library and framework authors, promote best practices on library and framework standardization, and improve the visibility and usability of our library ecosystem. This effort will only be possible with your active participation and invaluable feedback. So, we would like to invite you to share your understanding, expectations, and what you want to see from this working group. This discussion seeks to achieve two critical objectives: - *Gather Feedback on Your Understanding of the Group:* Your thoughts will help us determine if the group's mission and objectives are communicated and understood. That includes our short-term and long-term deliverables, our commitment to the community, and why we believe the Foundation's involvement is crucial in achieving our goals. We hope to refine our messaging based on your feedback, ensuring that our mission resonates with all. - *Understand Your Expectations and Interests:* What would you like to see us achieve? Would you like us to address any specific challenges you face as a developer or library author? Are there any areas of interest we should focus on? Your insights will guide our strategic planning and help us prioritize the most beneficial initiatives for the community. We value your opinions and would be grateful if you could take a few moments to share your thoughts. Your feedback is essential to ensure the Libraries and Frameworks Working Group meets the needs of the BEAM language ecosystem and serves the community in the best possible way. — Reply to this email directly, view it on GitHub <#1>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADRIVSXNFNGBMBDQW5SMLXNWP6PANCNFSM6AAAAAAZYWTTOQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Sent from my Mobile