Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Array.removeAt #4238

Open
delahee opened this issue May 20, 2015 · 37 comments
Open

Array.removeAt #4238

delahee opened this issue May 20, 2015 · 37 comments

Comments

@delahee
Copy link
Contributor

delahee commented May 20, 2015

Hi,

Deepnight and I think haxe Array misses :

function removeAt( pos : Int) : Void;

when there is only :

function splice( pos : Int, len : Int ) : Array<T>;

Splice is very problematique since int creates another array and totally trashes memory. On most platforms this function already exists so here is a big abstraction leak that haxe should fill.

for example on cpp, ths function would not ungrow the memory zone but use a memmove to retrieve the overlappig pointer zone, thus providing a very fast implementation.

So we propose we introduce this function in cross platform.

we can start with a naive implementation which is

function removeAt( pos : Int) : Void {
arr.remove( arr[idx] );
}

For informations, array is very often the (very) worst offender in memory usage yet the best ergonomic alternative, so we think improving it would be great.

@nadako
Copy link
Member

nadako commented May 20, 2015

I remember @waneck wanted to analyze unused splice results and generate spliceVoid method call that doesn"t create another array (that"s for C#/Java aka gencommon), but removeAt looks like a cleaner solution. However as always we"re stuck with native flash/js implementation and discussion of whether these additional methods should be available thru Dynamic and reflection (see #3776, #3323)

@Simn
Copy link
Member

Simn commented May 20, 2015

Modifying operations like this should be on Array itself so they can be properly optimized. That said it"s still going to require some effort to implement this and then there are still going to be issues if it is used through Dynamic, e.g. in templates.

I"m inclined to think that optimizing splice would be the better approach overall. In the static analyzer it would be easy to detect that its return value is not being used and then replace it with something the target generators can optimize.

@delahee
Copy link
Contributor Author

delahee commented May 20, 2015

@Simn I am not hot with the static analyzer option it feels too optionnal and avoiding this allocation should be mainstream.

I didn"t get that there was an issue with reflection, maybe its the good time to design a true cross platform solutions for extending Array ?

thx !

@ncannasse
Copy link
Member

I would NOT try to push too much index based add/remove on Array, since it"s a O(n) operation that can be quite inefficient compared to using a Map for instance

@delahee
Copy link
Contributor Author

delahee commented May 20, 2015

@ncannasse just a comment...your point is true only for big n, since it is cache aware and array allocation should be amortised, it is often faster to memmove than to use allocating + cache unaware tree for small n"s. Just like the same way insertion sort is faster than bubble sort for small n"s. In any case if it has native alternative removeAt will always be faster than splice and remove :)...

Anyway as a true side note, we would be happy enough if splice() could just not perform the allocation when possible, it would be sufficient for us.

thanks !

@ncannasse
Copy link
Member

I think that Array.remove does such thing already (not on Flash/JS since it uses splice)

@deltaluca
Copy link
Contributor

removeAt could not be implemented as arr.remove( arr[idx] ); as this would fail in general if the array have duplicates by equality, eg:

var xs = [0, 1, 0];
xs.remove(xs[2]);
trace(xs); // [1, 0]

Now, if you "did" consider that to be fine (order is not important in the Array), then you may aswell write implement as:

function removeAt(ind) {
    if (ind != length) this[ind] = pop();
}

Not that such a functino should be really named "removeAt" anymore.

@waneck
Copy link
Member

waneck commented Jun 3, 2015

I agree that going through the static analyzer and optimizing it when we can is probably the best solution. It"s not optional since it should be included by default on the next version of Haxe

@MSGhero
Copy link

MSGhero commented Jul 17, 2015

Flash is getting removeAt and insertAt, not sure if it"s a wrapper for splice or a whole new function: https://twitter.com/liquidate/status/621827033757126656

Edit: more here: http://labsdownload.adobe.com/pub/labs/flashruntimes/shared/air19_flashplayer19_releasenotes.pdf

@delahee
Copy link
Contributor Author

delahee commented Jul 17, 2015

Ahah that is a good lesson. We should stop being so conservative about such
things. :)

2015-07-17 3:34 GMT+02:00 Nick [email protected]:

Flash is getting removeAt and insertAt, not sure if it"s a wrapper for
splice or a whole new function:
https://twitter.com/liquidate/status/621827033757126656


Reply to this email directly or view it on GitHub
#4238 (comment)
.

David Elahee

@Justinfront
Copy link
Contributor

My day job is with as3 and there are a few nice things that surpise me, but Vectors are pretty crap, I just think Adobe should sit down with Haxe community and work something out. Haxe is pretty much as4+.

@delahee
Copy link
Contributor Author

delahee commented Sep 4, 2015

Ok now adobe made the thing, I"ll hack it into my haxe std libs and send a pull request.

@MSGhero
Copy link

MSGhero commented Sep 25, 2015

FP 19 was released earlier this week. Yay!

@delahee
Copy link
Contributor Author

delahee commented Sep 26, 2015

Yeah, definitive end of array removal that causes allocations !

@delahee
Copy link
Contributor Author

delahee commented Nov 8, 2015

ok work done, i"ll list pull request here.

This was referenced Nov 8, 2015
@delahee
Copy link
Contributor Author

delahee commented Nov 8, 2015

Pull requests done with tests, waiting for feedbacks :) thx !

@delahee
Copy link
Contributor Author

delahee commented Nov 8, 2015

Ok I think I know why the test fails, I ll put barriers in the test..

@andyli for the test to pass on cpp, hxcpp should be in sync with the PR...How should I proceed ?
Oh i found out there seems to be a build env problem...

@andyli
Copy link
Member

andyli commented Nov 8, 2015

If the PRs to both sides depend on each other, than there is no perfect way. My suggestion is to just ignore the build failure and check manually that the PRs are correct. Optionally, once both PRs are merged, restart the failed build to let it pick up the new commits from another side.

@delahee
Copy link
Contributor Author

delahee commented Nov 8, 2015

@andyli I think HaxeFoundation/hxcpp#328 is pretty much standalone, when it passes, #4633 will be ok with one fix.I"ll wait for you or @hughsando to check and accept HaxeFoundation/hxcpp#328 then we can move on, in the mean time, i"ll fix the utest tomorrow. thx.

@hughsando
Copy link
Member

So I have implemented this, but if your goal it to reduce allocations, then you should not have a return value, or allow it to be unspecified in the case of the index being out of range. Requiring a null return forces boxing of the result, even if it is ignored.
I would also like to suggest a "count" parameter, but then you might end up with the "is it count, or is it last" debacle we have with slice/splice.

@delahee
Copy link
Contributor Author

delahee commented Nov 17, 2015

Thanks you ! What is allocated then a Null ?

I would vote we thus remove the return parameters.

What do you think ?

@Justinfront
Copy link
Contributor

function removeAt( 5: Int, ?returnValue: Boolean = true ): T
I feel it"s important functionally and practically to return a null or the item removed by default, but it would be acceptable for it to return Void if returnValue was set to false.

@delahee
Copy link
Contributor Author

delahee commented Nov 18, 2015

This would be strange to have return on demand, feels like splice... this function was designed for allocation performance (otherwise splice is already here) so I think really prefer to remove any return so that we can keep best perfs...

@Justinfront
Copy link
Contributor

I think for haxe to implement a method with a less functional approach and inconsistantly with as3 purely for performance seems to be against Haxe principles such a function should be in an external haxelib only, I don"t think it should be added to the std if it does not implement a return.
http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/Array.html#removeAt()

@ncannasse
Copy link
Member

I agree we could remove the return.

@Justinfront
Copy link
Contributor

well so be it, but I expect it will cause some future confusion for haxe libraries used by as3.

@delahee
Copy link
Contributor Author

delahee commented Nov 18, 2015

We are aware of that but haxe have to choose what is good on most platforms, since we have splice and removeAt"s purpose is performance, we have to stick to our goals. If confusion there is we "ll explain in the function"s doc.

@hughsando
Copy link
Member

I think we can have the return value - as long as it is target dependent
when the index is out-of-range for numeric types.
I don"t think this is going to hurt the usefulness since the only real case
I can think of where you might want null would be:

while( (result= array.removeAt(0) != null )
result.destroy();

Hugh

On Thu, Nov 19, 2015 at 3:32 AM, delahee [email protected] wrote:

We are aware of that but haxe have to choose what is good on most
platforms, since we have splice and removeAt"s purpose is performance, we
have to stick to our goals. If confusion there is we "ll explain in the
function"s doc.


Reply to this email directly or view it on GitHub
#4238 (comment)
.

@delahee
Copy link
Contributor Author

delahee commented Jan 7, 2016

@hughsando then I propose we simply return a boolean if something was effectively removed ?
I"d like to squeeze this one these days.

@delahee
Copy link
Contributor Author

delahee commented Feb 22, 2016

For the record, in a IRc chat, @Simn proposed to think about something with culling call site allocations. Let"s see what comes around.

@Simn
Copy link
Member

Simn commented Feb 22, 2016

@hughsando: Do you think it would make sense to have non-allocating variants of some other Array methods which can be used for block-level calls? For instance, if we have { a.pop(); } on an Array<Int> we don"t have to deal with the Null<T> return because it doesn"t matter.

@Simn Simn modified the milestone: 3.3.0-rc1 Feb 23, 2016
@hughsando
Copy link
Member

I have been think about this too. Particularly splice, which creates a
whole array, that then gets discarded.
I would love a variant of map.get that does not need an allocation too. My
recent work on the CPP ast may be able to detect when this is assigned to,
say, an Int and call "getInt" instead. But it would be easier if it was
part of the Api.

On Mon, Feb 22, 2016 at 6:48 PM, Simon Krajewski [email protected]
wrote:

@hughsando https://github.com/hughsando: Do you think it would make
sense to have non-allocating variants of some other Array methods which can
be used for block-level calls? For instance, if we have { a.pop(); } on
an Array we don"t have to deal with the Null return because it
doesn"t matter.


Reply to this email directly or view it on GitHub
#4238 (comment)
.

@ncannasse
Copy link
Member

@hughsando would it not be more simple to change the way you return Null values in general? (using a struct on stack instead of heap ptr)

@nadako
Copy link
Member

nadako commented Feb 26, 2016

btw, c# target uses struct Null<T> so it"s stack allocated

@hughsando
Copy link
Member

Yes, that is possible. So is a Array of Null an array of structures of 12 bytes?
I have been thinking of using a Variant structure, like

string Variant
{
   int type;
   union
   {
        int ival;
        double dval;
       String sval;
       Dynamic dynamicVal;
   };
};

for dynamic function calling and some function returns. This would be 12/16 byes of 32/64 bit systems.

This is big change, and it is hard to see how to do it in such a was as to keep hxcpp compatible with earlier versions of haxe - although it should be possible.

@hughsando
Copy link
Member

As for easier, I"m hoping using getInt should be very easy.

@ncannasse
Copy link
Member

One quick hack would be that if there"s a method of the same name but with a prefix + type suffix, you can call it instead.

For instance .get() will be translated to .__hx__get_Int() for instance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.