The issue is that that random guy disappears and now the maintainers need to carry the burden to maintain the code.

That is why a standard library has higher standards and shouldn't accept bad code...

everyone wants to add new methods and then the file becomes unwieldy and hard to read

Go and read some of Python's standard library code... we're still ok over here, much better I'd say.

If one wants to add a method he can just define it outside of the type in a separate package

I don't know what you are pointing out as an issue here. Right now, you can define a function that takes in a type as its input and do as any struct's internal method would... Yes you can't extend that type's signature, but you can't in Python either since you need to inherit that type or compose it inside a field of another Class. Classes are something that Modular has been clear that is in sights, but that they want to wait to build things out properly at other lower levels first.

That is why a standard library has higher standards and shouldn't accept bad code

It's not even bad, just methods most people don't care and don't use.

Go and read some of Python's standard library code...

I did read some python code. If you happen to have read huggingface's diffusers source code you would see massive use of mixin classes and you get complex inheritance hierarchies. It's totally ok if you are fine with that. Python does not solve the expression problem per se. I would suggest you read some code from trait-based languages.

you can define a function that takes in a type as its input and do as any struct's internal method would...

I'm not sure what you mean here. You either gave up on generic code or on single dispatch.

you can't in Python either

Yes you can in Python. In case you don't know:

In [1]: class F:
   ...:     pass
   ...: 

In [2]: def foo(): print("haha")

In [3]: F.foo = foo

In [4]: F.foo()
haha

you need to inherit that type or compose it inside a field of another Class.

This is exactly the expression problem.

Classes are something that Modular has been clear that is in sights

Yes but I feel like modular favours traits more at the moment.

Yes you can in Python. In case you don't know:

That example lives at runtime and does not extend the type's signature. And it uses dynamic Class features: read this Mojo proposal

Python does not solve the expression problem per se

What problem ? If you want to extend a type's functionality you inherit or compose or build utility functions around it, that is how programming works when using an external library. Otherwise you could have conflicts in future redefinitions.

I would suggest you read some code from trait-based languages.

Rust does not permit defining external trait implementations for foreign types: Rust's coherence rules. You have to define your own internal trait and use that, which is to say: build your own abstractions if you want to extend the type's functionality

does not extend the type's signature.

It extends the type's signature at runtime.

In [6]: F.__dict__
Out[6]: 
mappingproxy({'__module__': '__main__',
              '__dict__': <attribute '__dict__' of 'F' objects>,
              '__weakref__': <attribute '__weakref__' of 'F' objects>,
              '__doc__': None,
              'foo': <function __main__.foo()>})

Unless you have a different definition of type signature?

that is how programming works

Bro trust me, I have heard more than a hundred times that someone tells me "this is how X works". If your mind is wired that way then I don't think we have room for discussion. But I understand where you are coming from.

Btw Swift-like trait extensions is clearly on modular's roadmap. I link it again here.

i'll also mention, I can definitely see the benefit in something like this, since I (and maybe others) already build small generic libraries for projects while things are still being worked out in the actual stdlib, then once a pr goes through, i switch things over to that. It lets me move a little faster, but i could also imagine doing the same things even if there was a more lax community library, though maybe to a lesser extent

I think you understood well the proposal indeed. About the "on our own terms", we could do it, but if the stdlib team doesn't redirect random new features to stdlib-extensions when new contributors are coming to mojo, we'll just cause more confusion about the process.

But if there is a good consensus among the top contributors of Mojo, we can do stdlib-extensions without Modular's blessing, and then come back for their blessings later on when we proved that this system is working well. In the meantime, some new features might slip by and be integrated directly to the stdlib without going through stdlib-extensions, but I guess it's not too bad.

We can wait for more feedback and decide when we know what the top contributors think about this.

I think an incubator might be a better name as extension might spark confusions with the technical capability of type extension.

Totally agree.

There is a technical question of how one would be able to iterate on top of a type already present in the standard library.

We'll see on a case by case basis. When it's adding new methods (a common use case), we can just convert that into a free functions.
If it's adding/modifying attributes, it would be too much hassle for stdlib-extensions it can go in the stdlib directly. If too much work is required to make the feature live outside the stdlib, it should go in the stdlib directly. For example the short string optimization feature isn't a good fit for stdlib-extensions because it changes everything in the String struct.

This will make it hard for the third party developer to depend on the types defined in the incubator, hence an incubator based package could be a very fragile thing.

Indeed. We shouldn't advertise this package as stable.

For example the short string optimization feature isn't a good fit for stdlib-extensions because it changes everything in the String struct.

I actually think it is the other way around. I think it would be better to recreate a String in the incubator with a special name and try to implement the small string optimisation there first. It is kind of what was done in C community AFAIK. After the incubator solution is proven and maybe even tried in some projects, it can be suggested to move to standard library, or graduate to it's own repo, in case that standard library went with another solution, but there are third parties which would rather go with the solution from the incubator.

Indeed this can be seen on a case by case basis and depends on the amount of additional work that one has to do to go through the incubator first. The Modular staff can also chim in and express if they want the feature quickly or not (meaning they'll allocate time to review it) and are confident about the design. Sometime it may not be 100% clear and there may be the need for a decision from the Modular staff, since they have good visibility on the internal roadmap.

Yep sorry I forgot about the shoutout, I had the idea of this proposal after looking at your repo :)

I think that would be very detrimental to the unity of the ecosystem, we need Modular's blessing and support.

Yeah, while I believe we can do it without Modular's blessing if the top contributors are on board, it will be indeed much better if modular is on board with it.

we need to decide if we want to take typing_extensions approach and offer backwards compatibility

Not even the stdlib have backward compatibility. There is no package manager in mojo, so I don't see how backward compatibility would help anybody. The target is to have a green CI on all commits of the main branch, and that should be good enough.

what kind of safety standards and EoL. schedule do we keep (patching old version's security vulnerabilities, mem. leaks), etc

I'd say that just like in the stdlib, everything is on a "best effort" basis. Mojo is not ready for production, nobody should use it in production, I don't think we can or should provide any commitment there.

Having said that, I think stdlib-extensions would be a better name

Frankly I don't mind too much about the name of the repo. I have no strong opinion on the subject. We can revert it to stdlib-extensions if you wish. I think you can discuss that with @mzaks who has an opinion about it too.

Thanks for voicing your concerns, they're legitimate :) I'll try to address them

so the maintainers may still get frequently pinged for questions like "is this change something that would ever get accepted into the stdlib anyways?

In a typical extensions/incubator repo, the core developers rarely/never ask the opinions of the maintainers of the main repo (here modularml/mojo). The questions the core developers ask themselves when doing a code review is "Can this be a good fit for the standard library?" If the answer is "yes" or "maybe", then the feature is accepted. If the answer is "no" the feature isn't accepted. Of course maintenance cost must be taken into account.

Now comes the question "What if the core developers were wrong? And the Modular staff has a different opinion about the feature?"

Well in most cases, the Modular staff won't have an opinion in the subject, because only very few new features are needed at any given time. So the likely answer they'll give is "We will maybe accept that in the future, we'll see". The stdlib-extensions repo is a good fit for this situation as it allows one to play with the feature, reache a high quality level, judge the popularity of the feature and be just one pull request away from being in the Mojo stdlib.

Then we have the situation where the core devs accepted a feature, but the stdlib team in the end decided that they don't want it. This happens all the time, and it means that it's time for this feature to go in a separate package and repo (when we have a package manager).
It's not because it's not in the stdlib that the code has no value. So in the end no code was lost. The code is one copy-paste away from being an independant package!

The last possible difficult situation is when the core developers reject a feature, but later on, the stdlib team says they want it.
Then no harm done. The core developers will acknowledge the decision. Then a second decision must be taken by the stdlib team: do they want the feature to go through stdlib-extensions first or can the contributor add the feature directly to the stdlib? They'll decide depending on the urgency, the size, the additional work required, the complexity, etc...

In the end, I don't think any of those scenarios will create frustration. Let me know you thoughts! Note that those scenarios are based on my experience working with incubators/extensions repos in Keras and Tensorflow. We can slightly change the process if we believe it's relevant.

Thanks for the thorough response! That does ease my worries a bit. I wonder if it might also be beneficial for the maintainers to keep a list of "things we want but can't prioritize internally at the moment" so we can have some sense of their current priorities as external contributors.

I agree with you, that would help!