Skip to content

Instantly share code, notes, and snippets.

@obfusk

obfusk/README.md

Last active December 27, 2024 06:54
Show Gist options
  • Save obfusk/61046e09cee352ae6dd109911534b12e to your computer and use it in GitHub Desktop.
Save obfusk/61046e09cee352ae6dd109911534b12e to your computer and use it in GitHub Desktop.
Download ZIP
baseline.profm not deterministic
Raw
README.md

Bug: baseline.profm not deterministic

See https://issuetracker.google.com/issues/231837768

Should be fixed by this commit, which is not yet part of an Android Gradle plugin release.

Analysis by @obfusk

As dump-baseline.py shows, this is another ordering issue; and sort-baseline.py can be used as a workaround.

$ cmp 1.profm 2.profm
1.profm 2.profm differ: byte 24, line 1
$ diff -Naur <( repro-apk dump-baseline 1.profm ) <( repro-apk dump-baseline 2.profm )
 profm version=002
 num_dex_files=2
 profile_idx=0
-  profile_key="classes2.dex"
-  num_type_ids=2418
-  num_class_ids=0
-profile_idx=1
   profile_key="classes.dex"
   num_type_ids=8122
   num_class_ids=738
+profile_idx=1
+  profile_key="classes2.dex"
+  num_type_ids=2418
+  num_class_ids=0
$ repro-apk sort-baseline 1.profm 1s.profm
$ cmp 1s.profm 2.profm && echo OK
OK

Code

The problem is that in ArtProfile.kt:

val profileData = HashMap<DexFile, DexFileData>()

Which means the order of iterating over profileData is non-deterministic.

In ArtProfileSerializer.kt, METADATA_FOR_N, V0_1_0_P, and V0_0_9_OMR1 sort before iterating:

profileData.entries.sortedBy { it.key.name }

And V0_0_5_O and V0_0_1_N sort as well:

for ((dex, data) in profileData.toSortedMap(DexFile)) {

But METADATA_0_0_2 doesn"t sort:

profileData.onEachIndexed { index, entry ->

And neither does V0_1_5_S:

profileData.forEach { entry ->

profileData.onEachIndexed { index, entry ->

profileData.onEachIndexed { index, entry ->

Which is why their output is non-deterministic.

Workaround proposed by @obfusk: sort baseline.profm

Using com.android.tools.profgen in build.gradle

// NB: Android Studio can"t find the imports; this does not affect the
// actual build since Gradle can find them just fine.

import com.android.tools.profgen.ArtProfileKt
import com.android.tools.profgen.ArtProfileSerializer
import com.android.tools.profgen.DexFile

project.afterEvaluate {
    tasks.each { task ->
        if (task.name.startsWith("compile") && task.name.endsWith("ReleaseArtProfile")) {
            task.doLast {
                outputs.files.each { file ->
                    if (file.name.endsWith(".profm")) {
                        println("Sorting ${file} ...")
                        def version = ArtProfileSerializer.valueOf("METADATA_0_0_2")
                        def profile = ArtProfileKt.ArtProfile(file)
                        def keys = new ArrayList(profile.profileData.keySet())
                        def sortedData = new LinkedHashMap()
                        Collections.sort keys, new DexFile.Companion()
                        keys.each { key -> sortedData[key] = profile.profileData[key] }
                        new FileOutputStream(file).with {
                            write(version.magicBytes$profgen)
                            write(version.versionBytes$profgen)
                            version.write$profgen(it, sortedData, "")
                        }
                    }
                }
            }
        }
    }
}

NB: this (ab)uses internal functions/values.

With build.gradle.kts

The internal functions are not accessible from Kotlin, but you can use the groovy code as a script plugin from build.gradle.kts:

// NB: when using build.gradle.kts instead of build.gradle, save this file as
// app/fix-profm.gradle and add the following line to app/build.gradle.kts:
//    apply(from = "fix-profm.gradle")

buildscript {
    repositories {
        google()
        mavenCentral()
    }
    dependencies {
        // NB: use this to specifiy the AGP version directly:
        // classpath "com.android.tools.build:gradle:7.4.0"
        // NB: use this with gradle/libs.versions.toml (and modify as needed):
        classpath "com.android.tools.build:gradle:${libs.versions.android.gradle.get()}"
    }
}

// NB: Android Studio can"t find the imports; this does not affect the
// actual build since Gradle can find them just fine.

import com.android.tools.profgen.ArtProfileKt
import com.android.tools.profgen.ArtProfileSerializer
import com.android.tools.profgen.DexFile

project.afterEvaluate {
    tasks.each { task ->
        if (task.name.startsWith("compile") && task.name.endsWith("ReleaseArtProfile")) {
            task.doLast {
                outputs.files.each { file ->
                    if (file.name.endsWith(".profm")) {
                        println("Sorting ${file} ...")
                        def version = ArtProfileSerializer.valueOf("METADATA_0_0_2")
                        def profile = ArtProfileKt.ArtProfile(file)
                        def keys = new ArrayList(profile.profileData.keySet())
                        def sortedData = new LinkedHashMap()
                        Collections.sort keys, new DexFile.Companion()
                        keys.each { key -> sortedData[key] = profile.profileData[key] }
                        new FileOutputStream(file).with {
                            write(version.magicBytes$profgen)
                            write(version.versionBytes$profgen)
                            version.write$profgen(it, sortedData, "")
                        }
                    }
                }
            }
        }
    }
}

Calling sort-baseline.py from build.gradle

Using sort-baseline.py from reproducible-apk-tools in build.gradle:

// NB: assumes reproducible-apk-tools is a submodule in the app repo"s
// root dir; adjust the path accordingly if it is found elsewhere
project.afterEvaluate {
    tasks.compileReleaseArtProfile.doLast {
        outputs.files.each { file ->
            if (file.toString().endsWith(".profm")) {
                exec {
                    commandLine(
                        "../reproducible-apk-tools/inplace-fix.py",
                        "sort-baseline", file
                    )
                }
            }
        }
    }
}

Fix proposed by @linsui: disable baseline profiles

Add this to build.gradle:

tasks.whenTaskAdded { task ->
    if (task.name.contains("ArtProfile")) {
        task.enabled = false
    }
}

or this to build.gradle.kts:

tasks.whenTaskAdded {
    if (name.contains("ArtProfile")) {
        enabled = false
    }
}

Build log

Before:

> Task :app:mergeReleaseArtProfile
> Task :app:compileReleaseArtProfile

After:

> Task :app:mergeReleaseArtProfile SKIPPED
> Task :app:compileReleaseArtProfile SKIPPED

APK differences

Removed from APK

assets/dexopt/baseline.prof
assets/dexopt/baseline.profm

Modified

NB: this is just one example we analysed, other apps may have other differences.

classes.dex
classes2.dex
classes3.dex
kotlin/internal/internal.kotlin_builtins
kotlin/kotlin.kotlin_builtins
kotlin/ranges/ranges.kotlin_builtins
kotlin/reflect/reflect.kotlin_builtins
Differences in .dex files

These differences seem to be an expected result of the changes.

Removed classes
Lkotlin/io/path/DirectoryEntriesReader;
Lkotlin/io/path/FileVisitorBuilder;
Lkotlin/io/path/FileVisitorImpl;
Lkotlin/io/path/LinkFollowing;
Lkotlin/io/path/PathNode;
Lkotlin/io/path/PathTreeWalkKt;
Lkotlin/io/path/PathWalkOption;
Lkotlin/io/path/FileVisitorBuilderImpl;
Lkotlin/io/path/PathTreeWalk;
Lkotlin/io/path/PathTreeWalk$bfsIterator$1;
Lkotlin/io/path/PathTreeWalk$dfsIterator$1;
Lkotlin/jvm/optionals/OptionalsKt;
Lkotlin/ranges/OpenEndRange$DefaultImpls;
Lkotlin/ranges/OpenEndRange;
Lkotlin/ranges/ComparableOpenEndRange;
Lkotlin/ranges/OpenEndDoubleRange;
Lkotlin/ranges/OpenEndFloatRange;
Removed methods
kotlin.io.path.DirectoryEntriesReader.<init>:(Z)V
kotlin.io.path.DirectoryEntriesReader.getFollowLinks:()Z
kotlin.io.path.DirectoryEntriesReader.preVisitDirectory:(Ljava/lang/Object;Ljava/nio/file/attribute/BasicFileAttributes;)Ljava/nio/file/FileVisitResult;
kotlin.io.path.DirectoryEntriesReader.preVisitDirectory:(Ljava/nio/file/Path;Ljava/nio/file/attribute/BasicFileAttributes;)Ljava/nio/file/FileVisitResult;
kotlin.io.path.DirectoryEntriesReader.readEntries:(Lkotlin/io/path/PathNode;)Ljava/util/List;
kotlin.io.path.DirectoryEntriesReader.visitFile:(Ljava/lang/Object;Ljava/nio/file/attribute/BasicFileAttributes;)Ljava/nio/file/FileVisitResult;
kotlin.io.path.DirectoryEntriesReader.visitFile:(Ljava/nio/file/Path;Ljava/nio/file/attribute/BasicFileAttributes;)Ljava/nio/file/FileVisitResult;
kotlin.io.path.FileVisitorImpl.<init>:(Lkotlin/jvm/functions/Function2;Lkotlin/jvm/functions/Function2;Lkotlin/jvm/functions/Function2;Lkotlin/jvm/functions/Function2;)V
kotlin.io.path.FileVisitorImpl.postVisitDirectory:(Ljava/lang/Object;Ljava/io/IOException;)Ljava/nio/file/FileVisitResult;
kotlin.io.path.FileVisitorImpl.postVisitDirectory:(Ljava/nio/file/Path;Ljava/io/IOException;)Ljava/nio/file/FileVisitResult;
kotlin.io.path.FileVisitorImpl.preVisitDirectory:(Ljava/lang/Object;Ljava/nio/file/attribute/BasicFileAttributes;)Ljava/nio/file/FileVisitResult;
kotlin.io.path.FileVisitorImpl.preVisitDirectory:(Ljava/nio/file/Path;Ljava/nio/file/attribute/BasicFileAttributes;)Ljava/nio/file/FileVisitResult;
kotlin.io.path.FileVisitorImpl.visitFile:(Ljava/lang/Object;Ljava/nio/file/attribute/BasicFileAttributes;)Ljava/nio/file/FileVisitResult;
kotlin.io.path.FileVisitorImpl.visitFile:(Ljava/nio/file/Path;Ljava/nio/file/attribute/BasicFileAttributes;)Ljava/nio/file/FileVisitResult;
kotlin.io.path.FileVisitorImpl.visitFileFailed:(Ljava/lang/Object;Ljava/io/IOException;)Ljava/nio/file/FileVisitResult;
kotlin.io.path.FileVisitorImpl.visitFileFailed:(Ljava/nio/file/Path;Ljava/io/IOException;)Ljava/nio/file/FileVisitResult;
kotlin.io.path.LinkFollowing.<clinit>:()V
kotlin.io.path.LinkFollowing.<init>:()V
kotlin.io.path.LinkFollowing.toLinkOptions:(Z)[Ljava/nio/file/LinkOption;
kotlin.io.path.LinkFollowing.toVisitOptions:(Z)Ljava/util/Set;
kotlin.io.path.PathNode.<init>:(Ljava/nio/file/Path;Ljava/lang/Object;Lkotlin/io/path/PathNode;)V
kotlin.io.path.PathNode.getContentIterator:()Ljava/util/Iterator;
kotlin.io.path.PathNode.getKey:()Ljava/lang/Object;
kotlin.io.path.PathNode.getParent:()Lkotlin/io/path/PathNode;
kotlin.io.path.PathNode.getPath:()Ljava/nio/file/Path;
kotlin.io.path.PathNode.setContentIterator:(Ljava/util/Iterator;)V
kotlin.io.path.PathTreeWalkKt.access$createsCycle:(Lkotlin/io/path/PathNode;)Z
kotlin.io.path.PathTreeWalkKt.access$keyOf:(Ljava/nio/file/Path;[Ljava/nio/file/LinkOption;)Ljava/lang/Object;
kotlin.io.path.PathTreeWalkKt.createsCycle:(Lkotlin/io/path/PathNode;)Z
kotlin.io.path.PathTreeWalkKt.keyOf:(Ljava/nio/file/Path;[Ljava/nio/file/LinkOption;)Ljava/lang/Object;
kotlin.io.path.PathWalkOption.$values:()[Lkotlin/io/path/PathWalkOption;
kotlin.io.path.PathWalkOption.<clinit>:()V
kotlin.io.path.PathWalkOption.<init>:(Ljava/lang/String;I)V
kotlin.io.path.PathWalkOption.valueOf:(Ljava/lang/String;)Lkotlin/io/path/PathWalkOption;
kotlin.io.path.PathWalkOption.values:()[Lkotlin/io/path/PathWalkOption;
kotlin.io.path.FileVisitorBuilderImpl.<init>:()V
kotlin.io.path.FileVisitorBuilderImpl.checkIsNotBuilt:()V
kotlin.io.path.FileVisitorBuilderImpl.checkNotDefined:(Ljava/lang/Object;Ljava/lang/String;)V
kotlin.io.path.FileVisitorBuilderImpl.build:()Ljava/nio/file/FileVisitor;
kotlin.io.path.FileVisitorBuilderImpl.onPostVisitDirectory:(Lkotlin/jvm/functions/Function2;)V
kotlin.io.path.FileVisitorBuilderImpl.onPreVisitDirectory:(Lkotlin/jvm/functions/Function2;)V
kotlin.io.path.FileVisitorBuilderImpl.onVisitFile:(Lkotlin/jvm/functions/Function2;)V
kotlin.io.path.FileVisitorBuilderImpl.onVisitFileFailed:(Lkotlin/jvm/functions/Function2;)V
kotlin.io.path.PathTreeWalk.<init>:(Ljava/nio/file/Path;[Lkotlin/io/path/PathWalkOption;)V
kotlin.io.path.PathTreeWalk.access$getFollowLinks:(Lkotlin/io/path/PathTreeWalk;)Z
kotlin.io.path.PathTreeWalk.access$getIncludeDirectories:(Lkotlin/io/path/PathTreeWalk;)Z
kotlin.io.path.PathTreeWalk.access$getLinkOptions:(Lkotlin/io/path/PathTreeWalk;)[Ljava/nio/file/LinkOption;
kotlin.io.path.PathTreeWalk.access$getStart$p:(Lkotlin/io/path/PathTreeWalk;)Ljava/nio/file/Path;
kotlin.io.path.PathTreeWalk.bfsIterator:()Ljava/util/Iterator;
kotlin.io.path.PathTreeWalk.dfsIterator:()Ljava/util/Iterator;
kotlin.io.path.PathTreeWalk.getFollowLinks:()Z
kotlin.io.path.PathTreeWalk.getIncludeDirectories:()Z
kotlin.io.path.PathTreeWalk.getLinkOptions:()[Ljava/nio/file/LinkOption;
kotlin.io.path.PathTreeWalk.isBFS:()Z
kotlin.io.path.PathTreeWalk.yieldIfNeeded:(Lkotlin/sequences/SequenceScope;Lkotlin/io/path/PathNode;Lkotlin/io/path/DirectoryEntriesReader;Lkotlin/jvm/functions/Function1;Lkotlin/coroutines/Continuation;)Ljava/lang/Object;
kotlin.io.path.PathTreeWalk.iterator:()Ljava/util/Iterator;
kotlin.io.path.PathsKt__PathUtilsKt.fileVisitor:(Lkotlin/jvm/functions/Function1;)Ljava/nio/file/FileVisitor;
kotlin.io.path.PathsKt__PathUtilsKt.visitFileTree:(Ljava/nio/file/Path;IZLkotlin/jvm/functions/Function1;)V
kotlin.io.path.PathsKt__PathUtilsKt.visitFileTree:(Ljava/nio/file/Path;Ljava/nio/file/FileVisitor;IZ)V
kotlin.io.path.PathsKt__PathUtilsKt.visitFileTree$default:(Ljava/nio/file/Path;IZLkotlin/jvm/functions/Function1;ILjava/lang/Object;)V
kotlin.io.path.PathsKt__PathUtilsKt.visitFileTree$default:(Ljava/nio/file/Path;Ljava/nio/file/FileVisitor;IZILjava/lang/Object;)V
kotlin.io.path.PathsKt__PathUtilsKt.walk:(Ljava/nio/file/Path;[Lkotlin/io/path/PathWalkOption;)Lkotlin/sequences/Sequence;
kotlin.io.path.PathTreeWalk$bfsIterator$1.<init>:(Lkotlin/io/path/PathTreeWalk;Lkotlin/coroutines/Continuation;)V
kotlin.io.path.PathTreeWalk$bfsIterator$1.create:(Ljava/lang/Object;Lkotlin/coroutines/Continuation;)Lkotlin/coroutines/Continuation;
kotlin.io.path.PathTreeWalk$bfsIterator$1.invoke:(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
kotlin.io.path.PathTreeWalk$bfsIterator$1.invoke:(Lkotlin/sequences/SequenceScope;Lkotlin/coroutines/Continuation;)Ljava/lang/Object;
kotlin.io.path.PathTreeWalk$bfsIterator$1.invokeSuspend:(Ljava/lang/Object;)Ljava/lang/Object;
kotlin.io.path.PathTreeWalk$dfsIterator$1.<init>:(Lkotlin/io/path/PathTreeWalk;Lkotlin/coroutines/Continuation;)V
kotlin.io.path.PathTreeWalk$dfsIterator$1.create:(Ljava/lang/Object;Lkotlin/coroutines/Continuation;)Lkotlin/coroutines/Continuation;
kotlin.io.path.PathTreeWalk$dfsIterator$1.invoke:(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
kotlin.io.path.PathTreeWalk$dfsIterator$1.invoke:(Lkotlin/sequences/SequenceScope;Lkotlin/coroutines/Continuation;)Ljava/lang/Object;
kotlin.io.path.PathTreeWalk$dfsIterator$1.invokeSuspend:(Ljava/lang/Object;)Ljava/lang/Object;
j$.util.Optional.orElse:(Ljava/lang/Object;)Ljava/lang/Object;
kotlin.jvm.optionals.OptionalsKt.asSequence:(Lj$/util/Optional;)Lkotlin/sequences/Sequence;
kotlin.jvm.optionals.OptionalsKt.getOrDefault:(Lj$/util/Optional;Ljava/lang/Object;)Ljava/lang/Object;
kotlin.jvm.optionals.OptionalsKt.getOrElse:(Lj$/util/Optional;Lkotlin/jvm/functions/Function0;)Ljava/lang/Object;
kotlin.jvm.optionals.OptionalsKt.getOrNull:(Lj$/util/Optional;)Ljava/lang/Object;
kotlin.jvm.optionals.OptionalsKt.toCollection:(Lj$/util/Optional;Ljava/util/Collection;)Ljava/util/Collection;
kotlin.jvm.optionals.OptionalsKt.toList:(Lj$/util/Optional;)Ljava/util/List;
kotlin.jvm.optionals.OptionalsKt.toSet:(Lj$/util/Optional;)Ljava/util/Set;
kotlin.ranges.OpenEndRange$DefaultImpls.contains:(Lkotlin/ranges/OpenEndRange;Ljava/lang/Comparable;)Z
kotlin.ranges.OpenEndRange$DefaultImpls.isEmpty:(Lkotlin/ranges/OpenEndRange;)Z
kotlin.ranges.RangesKt__RangesKt.contains:(Lkotlin/ranges/ClosedRange;Ljava/lang/Object;)Z
kotlin.ranges.RangesKt__RangesKt.contains:(Lkotlin/ranges/OpenEndRange;Ljava/lang/Object;)Z
kotlin.ranges.RangesKt__RangesKt.rangeUntil:(DD)Lkotlin/ranges/OpenEndRange;
kotlin.ranges.RangesKt__RangesKt.rangeUntil:(FF)Lkotlin/ranges/OpenEndRange;
kotlin.ranges.RangesKt__RangesKt.rangeUntil:(Ljava/lang/Comparable;Ljava/lang/Comparable;)Lkotlin/ranges/OpenEndRange;
kotlin.ranges.URangesKt___URangesKt.rangeUntil-5PvTz6A:(SS)Lkotlin/ranges/UIntRange;
kotlin.ranges.URangesKt___URangesKt.rangeUntil-J1ME1BU:(II)Lkotlin/ranges/UIntRange;
kotlin.ranges.URangesKt___URangesKt.rangeUntil-Kr8caGY:(BB)Lkotlin/ranges/UIntRange;
kotlin.ranges.URangesKt___URangesKt.rangeUntil-eb3DHEI:(JJ)Lkotlin/ranges/ULongRange;
kotlin.reflect.KCallable$DefaultImpls.getName$annotations:()V
kotlin.streams.jdk8.StreamsKt.$r8$lambda$D6rJ2g9z2pCQAEMFkqgtKPOz0JA:(Lkotlin/sequences/Sequence;)Lj$/util/Spliterator;
kotlin.streams.jdk8.StreamsKt.asStream$lambda$4:(Lkotlin/sequences/Sequence;)Lj$/util/Spliterator;
kotlin.math.MathKt__MathJVMKt.cbrt:(D)D
kotlin.math.MathKt__MathJVMKt.cbrt:(F)F
kotlin.ranges.ComparableOpenEndRange.<init>:(Ljava/lang/Comparable;Ljava/lang/Comparable;)V
kotlin.ranges.ComparableOpenEndRange.contains:(Ljava/lang/Comparable;)Z
kotlin.ranges.ComparableOpenEndRange.equals:(Ljava/lang/Object;)Z
kotlin.ranges.ComparableOpenEndRange.getEndExclusive:()Ljava/lang/Comparable;
kotlin.ranges.ComparableOpenEndRange.getStart:()Ljava/lang/Comparable;
kotlin.ranges.ComparableOpenEndRange.hashCode:()I
kotlin.ranges.ComparableOpenEndRange.isEmpty:()Z
kotlin.ranges.ComparableOpenEndRange.toString:()Ljava/lang/String;
kotlin.ranges.OpenEndDoubleRange.<init>:(DD)V
kotlin.ranges.OpenEndDoubleRange.lessThanOrEquals:(DD)Z
kotlin.ranges.OpenEndDoubleRange.contains:(D)Z
kotlin.ranges.OpenEndDoubleRange.contains:(Ljava/lang/Comparable;)Z
kotlin.ranges.OpenEndDoubleRange.equals:(Ljava/lang/Object;)Z
kotlin.ranges.OpenEndDoubleRange.getEndExclusive:()Ljava/lang/Comparable;
kotlin.ranges.OpenEndDoubleRange.getEndExclusive:()Ljava/lang/Double;
kotlin.ranges.OpenEndDoubleRange.getStart:()Ljava/lang/Comparable;
kotlin.ranges.OpenEndDoubleRange.getStart:()Ljava/lang/Double;
kotlin.ranges.OpenEndDoubleRange.hashCode:()I
kotlin.ranges.OpenEndDoubleRange.isEmpty:()Z
kotlin.ranges.OpenEndDoubleRange.toString:()Ljava/lang/String;
kotlin.ranges.OpenEndFloatRange.<init>:(FF)V
kotlin.ranges.OpenEndFloatRange.lessThanOrEquals:(FF)Z
kotlin.ranges.OpenEndFloatRange.contains:(F)Z
kotlin.ranges.OpenEndFloatRange.contains:(Ljava/lang/Comparable;)Z
kotlin.ranges.OpenEndFloatRange.equals:(Ljava/lang/Object;)Z
kotlin.ranges.OpenEndFloatRange.getEndExclusive:()Ljava/lang/Comparable;
kotlin.ranges.OpenEndFloatRange.getEndExclusive:()Ljava/lang/Float;
kotlin.ranges.OpenEndFloatRange.getStart:()Ljava/lang/Comparable;
kotlin.ranges.OpenEndFloatRange.getStart:()Ljava/lang/Float;
kotlin.ranges.OpenEndFloatRange.hashCode:()I
kotlin.ranges.OpenEndFloatRange.isEmpty:()Z
kotlin.ranges.OpenEndFloatRange.toString:()Ljava/lang/String;
kotlin.ranges.RangesKt___RangesKt.byteRangeContains:(Lkotlin/ranges/OpenEndRange;I)Z
kotlin.ranges.RangesKt___RangesKt.byteRangeContains:(Lkotlin/ranges/OpenEndRange;J)Z
kotlin.ranges.RangesKt___RangesKt.byteRangeContains:(Lkotlin/ranges/OpenEndRange;S)Z
kotlin.ranges.RangesKt___RangesKt.contains:(Lkotlin/ranges/IntRange;B)Z
kotlin.ranges.RangesKt___RangesKt.contains:(Lkotlin/ranges/IntRange;J)Z
kotlin.ranges.RangesKt___RangesKt.contains:(Lkotlin/ranges/IntRange;S)Z
kotlin.ranges.RangesKt___RangesKt.contains:(Lkotlin/ranges/LongRange;B)Z
kotlin.ranges.RangesKt___RangesKt.contains:(Lkotlin/ranges/LongRange;I)Z
kotlin.ranges.RangesKt___RangesKt.contains:(Lkotlin/ranges/LongRange;S)Z
kotlin.ranges.RangesKt___RangesKt.doubleRangeContains:(Lkotlin/ranges/OpenEndRange;F)Z
kotlin.ranges.RangesKt___RangesKt.intRangeContains:(Lkotlin/ranges/OpenEndRange;B)Z
kotlin.ranges.RangesKt___RangesKt.intRangeContains:(Lkotlin/ranges/OpenEndRange;J)Z
kotlin.ranges.RangesKt___RangesKt.intRangeContains:(Lkotlin/ranges/OpenEndRange;S)Z
kotlin.ranges.RangesKt___RangesKt.longRangeContains:(Lkotlin/ranges/OpenEndRange;B)Z
kotlin.ranges.RangesKt___RangesKt.longRangeContains:(Lkotlin/ranges/OpenEndRange;I)Z
kotlin.ranges.RangesKt___RangesKt.longRangeContains:(Lkotlin/ranges/OpenEndRange;S)Z
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(CC)Lkotlin/ranges/CharRange;
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(BB)Lkotlin/ranges/IntRange;
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(BI)Lkotlin/ranges/IntRange;
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(BS)Lkotlin/ranges/IntRange;
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(IB)Lkotlin/ranges/IntRange;
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(II)Lkotlin/ranges/IntRange;
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(IS)Lkotlin/ranges/IntRange;
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(SB)Lkotlin/ranges/IntRange;
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(SI)Lkotlin/ranges/IntRange;
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(SS)Lkotlin/ranges/IntRange;
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(BJ)Lkotlin/ranges/LongRange;
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(IJ)Lkotlin/ranges/LongRange;
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(JB)Lkotlin/ranges/LongRange;
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(JI)Lkotlin/ranges/LongRange;
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(JJ)Lkotlin/ranges/LongRange;
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(JS)Lkotlin/ranges/LongRange;
kotlin.ranges.RangesKt___RangesKt.rangeUntil:(SJ)Lkotlin/ranges/LongRange;
kotlin.ranges.RangesKt___RangesKt.shortRangeContains:(Lkotlin/ranges/OpenEndRange;B)Z
kotlin.ranges.RangesKt___RangesKt.shortRangeContains:(Lkotlin/ranges/OpenEndRange;I)Z
kotlin.ranges.RangesKt___RangesKt.shortRangeContains:(Lkotlin/ranges/OpenEndRange;J)Z
kotlin.ranges.CharRange.getEndExclusive$annotations:()V
kotlin.ranges.CharRange.getEndExclusive:()Ljava/lang/Character;
kotlin.ranges.CharRange.getEndExclusive:()Ljava/lang/Comparable;
kotlin.ranges.IntRange.getEndExclusive$annotations:()V
kotlin.ranges.IntRange.getEndExclusive:()Ljava/lang/Comparable;
kotlin.ranges.IntRange.getEndExclusive:()Ljava/lang/Integer;
kotlin.ranges.LongRange.getEndExclusive$annotations:()V
kotlin.ranges.LongRange.getEndExclusive:()Ljava/lang/Comparable;
kotlin.ranges.LongRange.getEndExclusive:()Ljava/lang/Long;
kotlin.ranges.UIntRange.getEndExclusive-pVg5ArA$annotations:()V
kotlin.ranges.UIntRange.getEndExclusive:()Ljava/lang/Comparable;
kotlin.ranges.UIntRange.getEndExclusive-pVg5ArA:()I
kotlin.ranges.ULongRange.getEndExclusive-s-VKNKU$annotations:()V
kotlin.ranges.ULongRange.getEndExclusive:()Ljava/lang/Comparable;
kotlin.ranges.ULongRange.getEndExclusive-s-VKNKU:()J
Added methods
kotlin.ranges.RangesKt__RangesKt.contains:(Ljava/lang/Iterable;Ljava/lang/Object;)Z
kotlin.streams.jdk8.StreamsKt.$r8$lambda$bdU4-xB_0bnfvMo-xyX7v8aTfMQ:(Lkotlin/sequences/Sequence;)Lj$/util/Spliterator;
kotlin.streams.jdk8.StreamsKt.asStream$lambda-4:(Lkotlin/sequences/Sequence;)Lj$/util/Spliterator;
Other changes

The procyon output shows various changes like this one:

     private static final Object[] copyToArrayImpl(final Collection collection, final Object[] ar

         Intrinsics.checkNotNullParameter((Object)collection, "collection");
         Intrinsics.checkNotNullParameter((Object)array, "array");
-        return CollectionToArray.toArray(collection, array);
+        final Object[] array2 = CollectionToArray.toArray(collection, array);
+        Intrinsics.checkNotNull((Object)array2, "null cannot be cast to non-null type kotlin.Array<T of kotlin.collections.CollectionsKt__CollectionsJVMKt.copyToArrayImpl>");
+        return array2;
     }
Differences in .kotlin_builtins files

These differences seem to be an expected result of the changes.

According to this stack overflow answer:

These files contain data for declarations of standard ("built-in") Kotlin classes which are not compiled to .class files, but rather are mapped to the existing types on the platform (in this case, JVM). For example, kotlin/kotlin.kotlin_builtins contains the information for non-physical classes in package kotlin: Int, String, Enum, Annotation, Collection, etc.

Performance impact: unknown

Repeating remarks from this issue:

I"m quite sure that disabling ART will give us a performance penalty, so we shouldn"t disable it.

It doesn"t disable the Android Runtime (ART), just baseline profiles.

According to the docs:

Baseline Profiles improve code execution speed by around 30% from the first launch by avoiding interpretation and just-in-time (JIT) compilation steps for included code paths.

So it may very well indeed result in a performance penalty, but I have found no real-world data on whether 30% is accurate or not. And all the docs say that developers need to define these profiles, but I don"t see e.g. a baseline-prof.txt in your repo, so I have no idea what the baseline.prof{,m} is currently based on and whether it"s actually doing anything useful. Do you have any info on that?

@obfusk
Copy link
Author

obfusk commented Jan 24, 2023

@jroddev @Yet-Zio FYI the workaround here -- sorting the .profm file in build.gradle -- might be a (better) solution for the .profm reproducibility issue affecting your F-Droid apps.

@penn5
Copy link

penn5 commented Jan 30, 2023

My app has differences in baseline.prof. Sorting the .profm doesn"t help.

To fix it I disabled baseline profiles. I prefer this code in build.gradle.kts as it doesn"t force all tasks to be evaluated, and it doesn"t guess the task names:

tasks.withType<CompileArtProfileTask>() {
    enabled = false
}

This prevents either baseline.prof or baseline.profm being created and my issue is solved.

@obfusk
Copy link
Author

obfusk commented Jan 30, 2023

My app has differences in baseline.prof

I"ve never seen differences in baseline.prof that were not the result of differences in one of the .dex files.

And if the .dex files are different, removing the .prof still doesn"t make the build reproducible.

@penn5 If your .dex files are identical, I"d love to see the .prof files (or a text dump of their contents).

@penn5
Copy link

penn5 commented Jan 30, 2023

Well, now you"ve got me. There was a difference in the dexes but I fixed it after dealing with the baseline.prof. I"ll see about some git reverts in the morning 🙃

@obfusk
Copy link
Author

obfusk commented Jan 30, 2023

For anyone interested in the details: the .prof file contains a checksum for each .dex file it contains information about; so if the .dex file differs, the .prof file will always differ as well.

Of course, not all .dex files may be represented in the .prof file, so it"s possible to have one of the .dex files differ with an identical .prof; but I"ve never seen a non-identical .prof w/ identical .dex.

It would be possible, if you have a .prof file in the 015 S serialisation format, since that had the same ordering bug as the .profm serialisation format 002.

But I have yet to see any of those "in the wild", which is also why dump-baseline.py does not support this format (yet).

@obfusk
Copy link
Author

obfusk commented Jan 31, 2023

The .profm sorting in build.gradle is now used by NewPipe: TeamNewPipe/NewPipe#9709

@licaon-kter
Copy link

but I"ve never seen a non-identical .prof w/ identical .dex

We"ve seen a case here: d4rken-org/sdmaid-se#1232 (comment)

My local APK: https://github.com/user-attachments/files/16430247/eu.darken.sdmse_10201000.apk.ZIP

Upstream: https://github.com/d4rken-org/sdmaid-se/releases/download/v1.2.1-rc0/eu.darken.sdmse-v1.2.1-rc0-10201000-FOSS-RELEASE.apk

@obfusk
Copy link
Author

obfusk commented Jul 30, 2024

@licaon-kter I very clearly told you:

But then you don"t get to ask for my help either. You"re on your own from now on. My issue trackers are closed to F-Droid now. Good luck next time.

This being a gist is a technicality. It doesn"t meaningfully change anything. Please do not force me to block you like Hans did.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment