Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stack overflow when deserialising int64 #47

Closed
csmager opened this issue May 29, 2018 · 16 comments
Closed

Stack overflow when deserialising int64 #47

csmager opened this issue May 29, 2018 · 16 comments

Comments

@csmager
Copy link

csmager commented May 29, 2018

This is a remarkably simple repro, but it took a long time to track down. I'd set up some endpoints and one kept killing the server process with StackOverflowException with no hint of what or where it was failing.

It turns out (possibly a separate issue) that the client will serialise a long / int64 like this:

{low: 5, high: 0, unsigned: false}

And deserialising this using Fable.Remoting.Json.FableJsonConverter will result in a StackOverflowException.

I've put together a small repro contrasting the behaviour of the converter here and the one in Fable.JsonConverter. The last 3 all cause the same issue.

printfn "Fable.JsonConverter"
parse<int> fable "\"5\""
parse<int> fable "\" 5\""
parse<int> fable "5"
parse<int> fable "\"text\""
parse<int64> fable "\"5\""
parse<int64> fable "\" 5\""
parse<int64> fable "5"
parse<int64> fable "\"text\""
parse<int64> fable "{low: 41, high: 0, unsigned: false}"

printfn "--"

printfn "Fable.Remoting"
parse<int> remoting "\"5\""
parse<int> remoting "\" 5\""
parse<int> remoting "5"
parse<int> remoting "\"text\""
parse<int64> remoting "\"5\""
parse<int64> remoting "\" 5\""
parse<int64> remoting "5"
parse<int64> remoting "\text\""
parse<int64> remoting "{low: 41, high: 0, unsigned: false}"

The output is

Fable.JsonConverter
Success: "5" -> 5
Success: " 5" -> 5
Success: 5 -> 5
Failed: "text", Could not convert string to integer: text. Path '', line 1, position 6.
Success: "5" -> 5L
Success: " 5" -> 5L
Success: 5 -> 5L
Failed: "text", Input string was not in a correct format.
Failed: {low: 41, high: 0, unsigned: false}, Expecting int64 but got StartObject
--
Fable.Remoting
Success: "5" -> 5
Success: " 5" -> 5
Success: 5 -> 5
Failed: "text", Could not convert string to integer: text. Path '', line 1, position 6.
Success: "5" -> 5L
Success: " 5" -> 5L
Process is terminating due to StackOverflowException.

I've tried (briefly) to work out where this is broken, but I've run out of time today.

@csmager
Copy link
Author

csmager commented May 29, 2018

Given it's a copy (albeit older) of the one in Fable, I searched there. Surprised it didn't come up when googling earlier! It's almost certainly this issue.

I guess the converter here could be updated to match the Fable version? I'm not sure if there are any changes specific to Remoting, however.

@Zaid-Ajaj
Copy link
Owner

@csmager thanks a lot for tracking this down, it seems to be that the custom converter needs an update to match what Fable is doing at the moment.

BTW are you using Fable 2.0 or Fable 1.x ?

I am trying to reproduce the issue with tests in this PR, I was unable to run the integration tests on my new windows box so I will let travis handle the work for me 😄

I guess the converter here could be updated to match the Fable version? I'm not sure if there are any changes specific to Remoting, however.

That's the plan, there is actually (so far) no specific changes for the fork of the converter, it is just that I thought that I can customize it whenever necessary

@csmager
Copy link
Author

csmager commented May 29, 2018

I'm using Fable 1.x. I too struggled to get the integration tests to run - either on Windows or macOS.

The specific trigger for this was actually a single case union with a long - something like

type SingleCase = SingleCase of int64

It was only later I discovered that the deserialisation results in a stack overflow for any conversion to int64 with invalid / unsupported content. It may be the javascript end only creates the wrong representation when part of a similar union. I can check it out tomorrow.

So I think there are two issues here:

  1. In some circumstance, the front-end will generate the wrong JSON for a long.
  2. The deserialiser will stack overflow when deserialising 'invalid' values to long (which is a duplicate of Stackoverflow trying to deserialize int64 fable-compiler/Fable#1306)

@Zaid-Ajaj
Copy link
Owner

Yes, I finally broke the tests 😄

All these tests run without issue:

QUnit.testCaseAsync "IServer.echoPrimitiveLong" <| fun test ->
    async {
        let! fstResult = server.echoPrimitiveLong (20L)
        let! sndResult = server.echoPrimitiveLong 0L
        let! thirdResult = server.echoPrimitiveLong -20L
        do test.equal true (fstResult = 20L)
        do test.equal true (sndResult = 0L)
        do test.equal true (thirdResult = -20L)
    }

QUnit.testCaseAsync "IServer.echoComplexLong" <| fun test -> 
    async {
        let input = { Value = 20L; OtherValue = 10 }
        let! output = server.echoComplexLong input 
        do test.equal true (input = output)
    }

QUnit.testCaseAsync "IServer.echoOptionalLong" <| fun test ->
    async {
        let! fstResult = server.echoOptionalLong (Some 20L)
        let! sndResult = server.echoOptionalLong None
        do test.equal true (fstResult = (Some 20L))
        do test.equal true (sndResult = None)
    }

but after the one with the single union case, I get the stackoverflow exception:

QUnit.testCaseAsync "IServer.echoSingleDULong" <| fun test ->
    async {
        let! output = server.echoSingleDULong (SingleLongCase 20L)
        do test.equal true (output = (SingleLongCase 20L))
    }

Now I have something to work with

@Zaid-Ajaj
Copy link
Owner

Zaid-Ajaj commented May 29, 2018

too struggled to get the integration tests to run - either on Windows or macOS

Can you please elaborate on what errors you got on both machines? on macOS it should have been as simple as ./build.sh IntegrationTests and on windows build.cmd IntegrationTests but I am having trouble with Selenium not finding the binary for some reason

In some circumstance, the front-end will generate the wrong JSON for a long.

Do you have a concrete example of such JSON so that I can add a test for it?

@csmager
Copy link
Author

csmager commented May 29, 2018

Can you please elaborate on what errors you got on both machines?

Yep, I'll run these again tomorrow and report back.

Do you have a concrete example of such JSON so that I can add a test for it?

This was essentially what was in my initial post, though it seems we've now narrowed the 'circumstances' to at least include a single case DU. When serialising that I would see this sort of representation in the browser dev tools for the failed request:

{ "SingleCase": {low: 5, high: 0, unsigned: false} }

And when you combine that with the bug in the deserialiser it means you get a stack overflow for these sort of requests.

@Nhowka
Copy link
Collaborator

Nhowka commented May 29, 2018

Could we use the Fable.JsonConverter package and if we need any special feature we override the relevant methods? Or that can cause some assembly mismatches?

@Zaid-Ajaj
Copy link
Owner

Just updating the code should be fine, the problem here seems to with the updated Fable.JsonConverter itself. It still cannot deserialize long in a single du, i.e. the test fails:

Tests Failed: 
[
    {
        "module": "Fable.Remoting",
        "name": "IServer.echoSingleDULong",
        "result": false,
        "message": "Test timed out",
        "actual": null,
        "expected": null,
        "testId": "54831a87",
        "negative": false,
        "runtime": 5001,
        "todo": false,
        "source": "internalStop/config.timeout<@https://code.jquery.com/qunit/qunit-2.3.2.js:1930:6"
    }
]

I will investigate this even more at the converter level, if converter round trip tests pass, then the problem must be with the serialization on the client

@Zaid-Ajaj
Copy link
Owner

I think we can confirm this is a Fable issue now (after updating the converter) not serializing int64 inside DU's correctly where the following code fails

open Fable.Core
open Fable.Core.JsInterop

type SingleCase  = SingleCase of int64

let input = SingleCase 20L
printfn "%A" (ofJson<SingleCase> (toJson input))

error:

Uncaught Error: Cannot deserialize into Test.SingleCase - Error (inflate): Input string was not in a correct format. - Data: '{"SingleCase":{"low":20,"high":0,"unsigned":false}}'

On the server I get the following error when I try to deserialize the data:

Expecting int64 but got StartObject

@csmager
Copy link
Author

csmager commented May 30, 2018

That makes sense. Fixing the converter here at least makes it easier to see where the problem is. Hopefully the related Fable client-side issue can be fixed at some point.

@csmager
Copy link
Author

csmager commented May 30, 2018

Re the integration tests, I'm fumbling in the dark a bit. I assume from the logging of 'Starting FireFox driver' that I need FireFox, so I've installed it. These are the results I get:

macOS

dotnet run
Starting Web server
Web server started
Getting server ready to listen for reqeusts
Server listening to requests
Starting FireFox Driver
[07:50:53 INF] Smooth! Suave listener started in 57.767 with binding 127.0.0.1:8080

Unhandled Exception: OpenQA.Selenium.WebDriverException: Cannot start the driver service on http://localhost:57959/
at OpenQA.Selenium.DriverService.Start()
at OpenQA.Selenium.Remote.DriverServiceCommandExecutor.Execute(Command commandToExecute)
at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
at OpenQA.Selenium.Remote.RemoteWebDriver.StartSession(ICapabilities desiredCapabilities)
at OpenQA.Selenium.Remote.RemoteWebDriver..ctor(ICommandExecutor commandExecutor, ICapabilities desiredCapabilities)
at Program.main(String[] argv) in /Users/cmager/dev/Fable.Remoting/UITests/Program.fs:line 98

Windows

NPM fails to download the node-sass 4.7.2 binary. On looking through the project's releases, I noticed that this version doesn't support Node 10. I think it generates a file name for the binary that includes a node version number based on this mapping. As this version doesn't support Node 10, the file name it generates doesn't include any node version and that file doesn't exist.

Running npm update node-sass --save gets me further, but it ends here:

cmd /C dotnet run
Starting Web server
Web server started
Getting server ready to listen for reqeusts
Server listening to requests
Starting FireFox Driver
[08:07:04 INF] Smooth! Suave listener started in 55.923 with binding 127.0.0.1
:8080
Unhandled Exception: OpenQA.Selenium.DriverServiceNotFoundException: The file C:\dev\Fable.Remoting\UITests\drivers\geckodriver.exe does not exist. The driver can be downloaded at https://github.com/mozilla/geckodriver/releases
at OpenQA.Selenium.DriverService..ctor(String servicePath, Int32 port, String driverServiceExecutableName, Uri driverServiceDownloadUrl)
at OpenQA.Selenium.Firefox.FirefoxDriver..ctor(String geckoDriverDirectory, FirefoxOptions options)
at Program.main(String[] argv) in C:\dev\Fable.Remoting\UITests\Program.fs:line 98

@csmager
Copy link
Author

csmager commented May 30, 2018

Updating to the latest geckodriver/ geckodriver.exe fixes it for both.

@Zaid-Ajaj
Copy link
Owner

@csmager Thanks for the repro! In the Long serialization PR, I did both things too, I removed the node-sass dependency (I have no idea why it was there 😂) and updated the latest windows geckodriver.

@Zaid-Ajaj
Copy link
Owner

Fixed and merged. published nuget packages should be available soon, can you please update and confirm the that it is fixed?

@csmager
Copy link
Author

csmager commented Jun 5, 2018

@Zaid-Ajaj sorry it's taken a few days, I've only just got back to what I was working on previously. I reverted the workaround I had and it all seems to work now. Thanks!

@Zaid-Ajaj
Copy link
Owner

Awesome! thanks for the reporting back 😄 I will close this issue, please let me know if it persists in some other weird edge case

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants