Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

First configuration to resolve determines effective dynamic version cache policy #3019

Closed
DanielThomas opened this issue Sep 25, 2017 · 1 comment
Closed

First configuration to resolve determines effective dynamic version cache policy #3019

DanielThomas opened this issue Sep 25, 2017 · 1 comment
Assignees
@bigdaz
Labels
a:bug
Milestone
4.6 RC1

Comments

@DanielThomas
Copy link
Contributor

When a dynamic version is first resolved, that version list cached for the duration of the build. However, if the first configuration to resolve doesn't carry the desired cacheDynamicVersionsFor setting (as is the case for detached configurations), that will lead to stale dependencies being selected.

Expected Behavior

The individual cache TTL is honored.

Current Behavior

The cache TTL of the first configuration to resolve determines the staleness of dynamic dependencies.

Context

When we update dependency locks, we set changing/dynamic caching to 0, however plugins like Spring Dependency Management use detached configurations as part of dependency resolution, causing that configuration to be ignored.

Steps to Reproduce (for bugs)

Build

Run gw resolve --debug | grep DynamicVersion -A5 against the following build:

plugins {
	id 'java'
}

repositories {
	mavenCentral()
}

dependencies {
	compile 'com.google.guava:guava:latest.release'
}

configurations.all {
	resolutionStrategy {
	 	cacheDynamicVersionsFor 0, 'seconds'
	}
}

task resolve() {
	doFirst {
		// Resolve a configuration that doesn't have the desired dynamic TTL
		configurations.detachedConfiguration(dependencies.create('com.google.guava:guava:latest.release')).files

		// Resolve the desired configuration
		configurations.compile.files
	}
}

Observation

Note that the first dynamic version lookup indicates Using cached module metadata and the following requests perform no lookup:

14:48:42.749 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.DynamicVersionResolver] Attempting to resolve version for com.google.guava:guava:latest.release using repositories [MavenRepo]
14:48:42.749 [INFO] [org.gradle.cache.internal.DefaultCacheAccess] Creating new cache for metadata-2.23/module-versions, path /Users/dannyt/.gradle/caches/modules-2/metadata-2.23/module-versions.bin, access org.gradle.cache.internal.DefaultCacheAccess@1d5f484
14:48:42.750 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Opening cache module-versions.bin (/Users/dannyt/.gradle/caches/modules-2/metadata-2.23/module-versions.bin)
14:48:42.750 [INFO] [org.gradle.cache.internal.DefaultCacheAccess] Creating new cache for metadata-2.23/module-metadata, path /Users/dannyt/.gradle/caches/modules-2/metadata-2.23/module-metadata.bin, access org.gradle.cache.internal.DefaultCacheAccess@1d5f484
14:48:42.750 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Opening cache module-metadata.bin (/Users/dannyt/.gradle/caches/modules-2/metadata-2.23/module-metadata.bin)
14:48:42.751 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.CachingModuleComponentRepository] Using cached module metadata for module 'com.google.guava:guava:23.0' in 'MavenRepo'
--
14:48:42.751 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.DynamicVersionResolver] Using com.google.guava:guava:23.0 from Maven repository 'MavenRepo'
14:48:42.751 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Selecting new module version com.google.guava:guava:23.0
14:48:42.751 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Visiting configuration com.google.guava:guava:23.0(default).
14:48:42.751 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Visiting dependency com.google.guava:guava:23.0(default) -> dependency: com.google.code.findbugs:jsr305:1.3.9, scope: Compile, optional: false
14:48:42.751 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Selecting new module version com.google.code.findbugs:jsr305:1.3.9
14:48:42.751 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.RepositoryChainComponentMetaDataResolver] Attempting to resolve component for com.google.code.findbugs:jsr305:1.3.9 using repositories [MavenRepo]
--
14:48:42.757 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.DynamicVersionResolver] Attempting to resolve version for com.google.guava:guava:latest.release using repositories [MavenRepo]
14:48:42.758 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.DynamicVersionResolver] Using com.google.guava:guava:23.0 from Maven repository 'MavenRepo'
14:48:42.758 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Selecting new module version com.google.guava:guava:23.0
14:48:42.758 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Visiting configuration com.google.guava:guava:23.0(default).
14:48:42.758 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Visiting dependency com.google.guava:guava:23.0(default) -> dependency: com.google.code.findbugs:jsr305:1.3.9, scope: Compile, optional: false
14:48:42.758 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Selecting new module version com.google.code.findbugs:jsr305:1.3.9
14:48:42.758 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.RepositoryChainComponentMetaDataResolver] Attempting to resolve component for com.google.code.findbugs:jsr305:1.3.9 using repositories [MavenRepo]

Your Environment

Gradle 3.5.1
@bigdaz
Copy link
Member

bigdaz commented Dec 24, 2017

Thanks @DanielThomas for tracking this down.

This is a pretty nasty issue, caused by our in-memory resolution cache not respecting the different cache expiry settings for different configurations.
A similar problem exists where Maven SNAPSHOT modules will not be updated when they are first resolved by a detached configuration (that misses any cache expiry settings).

I hope to have a fix ready for 4.5.

@bigdaz bigdaz self-assigned this Dec 31, 2017
@bigdaz bigdaz added this to the 4.6 RC1 milestone Jan 5, 2018
@bigdaz bigdaz mentioned this issue Jan 17, 2018
Closed
@bigdaz bigdaz mentioned this issue Feb 4, 2018
Closed
@wolfs wolfs mentioned this issue Feb 5, 2018
Closed
bigdaz added a commit that referenced this issue Feb 5, 2018
@bigdaz
Add tests demonstrating #3019
f0c110c 
These tests demonstrate that Gradle only considers the cache expiry
policy for the first configuration that resolves a particular
dynamic version or snapshot module. This is particularly problematic
since detached configurations never get any cache expiry configured
via `configurations.all`.

As such, if a detached configuration is the first to resolve a dynamic
version or snapshot during a build invocation, it may cause the user-
configured cache expiry to be ignored. Several popular plugins,
including the Spring dependency management plugin, add detached
configurations that will be resolved early in a build invocation.
bigdaz added a commit that referenced this issue Feb 5, 2018
@bigdaz
Add failing test for #4261
2a0c2e2 
In-memory caching of module metadata is done on a per-repository
basis: the same repository definition in different projects will
allow sharing of in-memory cached module metadata.

Component metadata rules are defined on a per-project basis. This
means that in the scenario where different component metadata rules
have been defined for different projects, only the rules from the
first project to resolve a particular module from a particular
repository will apply.

This fix for this issue will require further changes beyond the fix
for #3019.
bigdaz added a commit that referenced this issue Feb 5, 2018
@bigdaz
Simple fix for cache-expiry by removing in-memory cache
06ef173 
This fix demonstrates that the incorrect dynamic version behaviour
of #3019 is due to the in-memory cache of module version lists.
A more complete fix will involve moving the in-memory caching to live
with the other filesystem caching, allowing the cache expiry policy
to be correctly honoured.
bigdaz added a commit that referenced this issue Feb 5, 2018
@bigdaz
Cache processed metadata in-memory
6e595ea 
Processing component metadata rules for resolved module metadata
is expensive, and the fix for #3019 introduced a serious performance
regression. This change adds in-memory caching for the post-processed
metadata on a per-repository basis.

Unfortunately, this re-introduces the bug #4261, where component
metadata rule outputs can leak between projects for a repository.
Since defining different metadata rules in different projects is
unlikely to be commonplace, this is considered an acceptable
interim position. This issue should be addressed with the
introduction of general-purpose caching of component metadata
rules.
@bigdaz bigdaz closed this as completed in b761585 Feb 6, 2018
bigdaz added a commit that referenced this issue Feb 6, 2018
@bigdaz
Merge pull request #4265 from gradle/dd/issues/3019
d10d340 
Fixes #3019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bigdaz bigdaz

Labels
a:bug
Projects
None yet
Milestone
4.6 RC1
Development

No branches or pull requests
3 participants
@bigdaz @bmuschko @DanielThomas