Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slow when multiple repos #1085

Closed
vchekan opened this issue Sep 24, 2015 · 12 comments
Closed

Slow when multiple repos #1085

vchekan opened this issue Sep 24, 2015 · 12 comments

Comments

@vchekan
Copy link

vchekan commented Sep 24, 2015

When I have many repositories, paket works quite slow because it seems to me, paket does evaluation of every source with all supported protocols until one response is successful.

The worst case I experienced, was converting from nuget of a large project, which use our internal nuget repo mirror of nuget.org and another one, based on CI server. In process of conversion, paket ignored disable section of nuget.config, and added back nuget.org for both v2 and v3. Thus, import ended up with 4 servers, nuget v2,v3 and 2 of our internal ones. Repeating every protocol for every repository for every package in large project took >1hr :(

I'm thinking, would it be possible to have lazily evaluated member in repository object, which would probe for supported version upon first call and than use only the successful protocol for given repository? This way it will eliminate a lot of failed http requests and save time.

@forki
Copy link
Member

forki commented Sep 29, 2015

For the convert-from-nuget issue I created #1098

Regarding feed queries: we query all configured feeds that are written above a nuget package line in the dependencies file. An easy workaround is the following:

source http://fastfeed

nuget FastPackage1 // looks only into http://fastfeed
nuget FastPackage2 // looks only into http://fastfeed

source http://slowfeed
nuget SlowPackage1 // loks into http://slowfeed and http://fastfeed

"which would probe for supported version upon first call and than use only the successful protocol for given repository"

yes that is some I have planned for long time. Hope to implement that soon

@forki
Copy link
Member

forki commented Sep 29, 2015

first results are coming in. 2.5.0-alpha002 is 4.2x faster on such a scenario with multiple feeds. try it out

@vchekan
Copy link
Author

vchekan commented Sep 29, 2015

I've tried e317a91. Marginally faster. Still 55min.
For some reason it keeps trying to interpret v3 json as xml.

{
  "@id": "https://www.nuget.org/packages/{id}/{version}/ReportAbuse",
  "@type": "ReportAbuseUriTemplate/3.0.0-beta",
  "comment": "URI template used by NuGet Client to construct Report Abuse URL for packages"
},
{
  "@id": "https://api.nuget.org/v3/stats0/totals.json",
  "@type": "TotalStats/3.0.0-rc",
  "comment": "Endpoint to get stats totals to display in nuget.org home page"
}

],
"@context": {
"@vocab": "http://schema.nuget.org/services#",
"comment": "http://www.w3.org/2000/01/rdf-schema#comment"
}
}
Source 'https://api.nuget.org/v3/index.json' exception: System.Xml.XmlException: Data at the root level is invalid. Line 1, positi
on 1.
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.LoadXml(String xml)
at Paket.NuGetV2.parseODataDetails[a](String nugetURL, PackageName packageName, a version, String raw) in C:\funprojects\Paket
src\Paket.Core\NuGetV2.fs:line 183
at [email protected](Unit unitVar) in C:\funprojects\Paket\src\Paket.Core\NuGetV2.fs:line
284
at [email protected](AsyncParams1 args) --- End of stack trace from previous location where exception was thrown --- at Microsoft.FSharp.Control.AsyncBuilderImpl.commit[a](Result1 res)
at Microsoft.FSharp.Control.CancellationTokenOps.RunSynchronously[a](CancellationToken token, FSharpAsync1 computation, FSharp Option1 timeout)
at Microsoft.FSharp.Control.FSharpAsync.RunSynchronously[T](FSharpAsync1 computation, FSharpOption1 timeout, FSharpOption1 c ancellationToken) at Paket.NuGetV2.tryNext@622(String root, Boolean force, FSharpList1 sources, PackageName packageName, SemVerInfo version, FSh
arpList`1 xs) in C:\funprojects\Paket\src\Paket.Core\NuGetV2.fs:line 629

Trying source 'https://www.nuget.org/api/v2'

@forki
Copy link
Member

forki commented Sep 30, 2015

could you please mail me your dependencies and lock files? sforkmann @ gmail

1h update is very very unusal. Maybe I can find the issue.

forki added a commit that referenced this issue Sep 30, 2015
@forki
Copy link
Member

forki commented Sep 30, 2015

latest version runs all calls against different NuGet protocols in parallel and takes the fastest response. But please still send the dependencies and lock files.

@vchekan
Copy link
Author

vchekan commented Sep 30, 2015

It is convert-from-nuget command.
Here is initial nuget.config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <config>
       <add key="repositorypath" value="$(Solutiondir)\..\packages" />
    </config>

    <disabledPackageSources>
         <add key="nuget.org" value="true" />
         <add key="mirror" value="true" />
         <add key="NTent" value="true" />
    </disabledPackageSources>

    <packageSources>
      <add key="nuget-proxy" value="http://localhost:8081/nexus/service/local/nuget/ntent/" />
    </packageSources>
</configuration>

As you see, all sources except nexus is disabled
When I run import, it generate paket.dependencies:

source http://ci-server.cb.ntent.com/guestAuth/app/nuget/v1/FeedService.svc/
source https://api.nuget.org/v3/index.json
source https://www.nuget.org/api/v2/
source http://localhost:8081/nexus/service/local/nuget/ntent/

nuget angularjs 1.4.3 framework: >= net45
nuget AngularTemplates.Compile 1.0.0 framework: >= net45
nuget Antlr 3.4.1.9004 framework: >= net45
nuget Autofac 3.5.0 framework: >= net45
nuget Autofac.Owin 3.1.0 framework: >= net45
nuget Autofac.WebApi 3.1.0 framework: >= net45
nuget Autofac.WebApi2 3.4.0 framework: >= net45
nuget Autofac.WebApi2.Owin 3.2.0 framework: >= net45
...
93 lines, 15 of which is private packages on local repo

4 "source" is more than I'd like to have, and because it is created during the import, there is no opportunity to edit them out until import is complete :(

I am going to use Sonatype's Nexus server, which is more known for Maven support, but it has very good nuget server implementation, in particular, so called "group server", which is a proxy which will do requests to multiple sources and cache result. So it would be extremely benefitial to avoid requests to disabled servers.

BTW, I'm at today's build, 8e4a0e3

Is there any way to redirect output to a text file? Import from nuget will ask for some confirmations on console which makes it impossible to redirect output :(

@forki
Copy link
Member

forki commented Oct 1, 2015

4 "source" is more than I'd like to have, and because it is created during the import, there is no opportunity to edit them out until import is complete :

ah so you are complaining about the convert-from-nuget performance? fortunately that is a step that you only do once. After that you want to edit your paket.dependencies file and remove unneeded sources. you also might want to run simplify and relax some of the version restrictions.
If you do that, then what is the runtime for paket update?

(For the convert-from-nuget issue I created #1098 and @theimowski will take a look)

@theimowski
Copy link
Member

@vchekan do I understand correctly that those feeds listed in disabled section are defined in different, higher-level nuget.config file and are identified by the key attribute?

@vchekan
Copy link
Author

vchekan commented Oct 1, 2015

@theimowski yes, that's correct.

@theimowski
Copy link
Member

Ok I think I know how it could be fixed. Gonna try send a fix this weekend

@forki
Copy link
Member

forki commented Oct 3, 2015

@vchekan Paket 2.7.2 has fixed convert-from-nuget. Thanks to @theimowski

@forki forki closed this as completed Oct 9, 2015
@vchekan
Copy link
Author

vchekan commented Nov 6, 2015

Measured my benchmark project, it takes now 2 minutes and 8 seconds to import a project which took 1hr before. Great job!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants