build: Allow to override build date #3417

bmwiedemann · 2024-02-11T15:02:43Z

Allow to override build date with SOURCE_DATE_EPOCH because
in builds from a release tarball, there is no git commit to take a date from, so the current date was inserted there.

See https://reproducible-builds.org/ for why this matters and https://reproducible-builds.org/specs/source-date-epoch/ for the definition of this variable.

This patch was done while working on reproducible builds for openSUSE.

note: I only tested the build and only on 8.2.1

in builds from a release tarball, there is no git commit to take a date from, so the current date was inserted there. See https://reproducible-builds.org/ for why this matters and https://reproducible-builds.org/specs/source-date-epoch/ for the definition of this variable. This patch was done while working on reproducible builds for openSUSE.

echoix · 2024-02-11T15:26:19Z

I"ll take a closer look later, but I"m really favorable on the concept. Thanks!

utils/mkhtml.py

tmszi · 2024-02-13T05:13:58Z

When GRASS GIS is compiled from the release tarball source code (without Git, and without official patch https://github.com/OSGeo/grass/releases/download/8.3.1/core_modules_with_last_commit.patch) getting date is provide with this function get_default_git_log() (line 218) :

grass/utils/mkhtml.py

Lines 407 to 409 in aaecfc8

    
           # During GRASS GIS compilation from source code without Git 
        
           else: 
        
               return get_git_commit_from_file(src_dir=src_dir)

grass/utils/mkhtml.py

Lines 259 to 261 in aaecfc8

    
           # Accessed date time if getting commit from JSON file wasn"t successful 
        
           if not git_log: 
        
               git_log = get_default_git_log(src_dir=src_dir)

grass/utils/mkhtml.py

Lines 205 to 221 in aaecfc8

    
           def get_default_git_log(src_dir, datetime_format="%A %b %d %H:%M:%S %Y"): 
        
               """Get default Git commit and commit date, when getting commit from 
        
               local Git, local JSON file and remote GitHub REST API server wasn"t 
        
               successful. 
        
               :param str src_dir: addon source dir 
        
               :param str datetime_format: output commit datetime format 
        
                                           e.g. Sunday Jan 16 23:09:35 2022 
        
               :return dict: dict which store last commit and commnit date 
        
               """ 
        
               return { 
        
                   "commit": "unknown", 
        
                   "date": datetime.fromtimestamp(os.path.getmtime(src_dir)).strftime( 
        
                       datetime_format 
        
                   ), 
        
               }

Compilation was tested with GRASS GIS official release tarball source code 8.3.1 version under Gentoo GNU/Linux distribution.

r.surf.rst module man page example:

tmszi · 2024-02-13T05:17:37Z

Ok I see patch is for reproducible builds for openSUSE distribution.

tmszi · 2024-02-13T05:43:50Z

Small note:

You can use official patch core_modules_with_last_commit.patch or simple
core_modules_with_last_commit.json file for getting commits for GRASS GIS core modules for compilation under openSUSE distribution.

Patch/JSON file is generated for every official release, you can find them under release URL https://github.com/OSGeo/grass/releases page under Assets section.

bmwiedemann · 2024-02-13T07:20:21Z

Unfortunately we are still on 8.2.1 atm and there is no https://github.com/OSGeo/grass/releases/download/8.2.1/core_modules_with_last_commit.json / patch - is there a way to only take my code-path for when that file is missing?

tmszi · 2024-02-13T07:55:14Z

Unfortunately we are still on 8.2.1 atm and there is no https://github.com/OSGeo/grass/releases/download/8.2.1/core_modules_with_last_commit.json / patch - is there a way to only take my code-path for when that file is missing?

Yes you are right, patch/JSON file with core modules commits are generated for version >= 8.3.

Doesn"t return this line code datetime.fromtimestamp(os.path.getmtime(src_dir)) (code return the time of last modification of path of source code dir) valid date time during reproducible build process from release tarball for openSUSE distro?

neteler · 2024-02-13T08:50:44Z

Yes you are right, patch/JSON file with core modules commits are generated for version >= 8.3.

@tmszi Could that be backported to 8.2?

tmszi · 2024-02-13T11:48:06Z

Yes you are right, patch/JSON file with core modules commits are generated for version >= 8.3.

@tmszi Could that be backported to 8.2?

I think so, but I will have to look into the source code (and prepare backport PR) to give you the exact answer.

tmszi · 2024-02-13T19:10:34Z

Yes you are right, patch/JSON file with core modules commits are generated for version >= 8.3.

@tmszi Could that be backported to 8.2?

The source code that is needed to automatically generate core modules with the last commit JSON file and patch file is included in the realeasebranch_8_2 branch.

I"m wonder why these files is not automatically created for 8.2.x realease. GitHub workflow config files looks good to me.

grass/.github/workflows/create_release_draft.yml

Lines 34 to 38 in 0bd5c8d

    
                 - name: Generate core modules with last commit JSON file and patch file 
        
                   run: | 
        
                     python utils/generate_last_commit_file.py . 
        
                     git add core_modules_with_last_commit.json 
        
                     git diff --cached > core_modules_with_last_commit.patch

grass/.github/workflows/create_release_draft.yml

Lines 62 to 71 in 0bd5c8d

    
           - name: Upload core_modules_with_last_commit.json file (for tags only) 
        
             if: startsWith(github.ref, "refs/tags/") 
        
             uses: actions/upload-release-asset@v1 
        
             env: 
        
               GITHUB_TOKEN: ${{ github.token }} 
        
             with: 
        
               upload_url: ${{ steps.create_release.outputs.upload_url }} 
        
               asset_path: core_modules_with_last_commit.json 
        
               asset_name: core_modules_with_last_commit.json 
        
               asset_content_type: application/json

grass/.github/workflows/create_release_draft.yml

Lines 95 to 104 in 0bd5c8d

    
                 - name: Make the created files available 
        
                   uses: actions/upload-artifact@v3 
        
                   if: always() 
        
                   with: 
        
                     name: artifacts 
        
                     path: | 
        
                       core_modules_with_last_commit.* 
        
                       ChangeLog.gz 
        
                     if-no-files-found: warn 
        
                     retention-days: 10

tmszi · 2024-02-13T19:27:48Z

@bmwiedemann It is possible generate JSON/patch file manually for 8.2.1 version:

git clone -b relasebranch_8_2 --single-branch https://github.com/OSGeo/grass.git && cd grass/
git checkout tags/8.2.1
python utils/generate_last_commit_file.py .
file core_modules_with_last_commit.json
git add core_modules_with_last_commit.json
git diff --cached > core_modules_with_last_commit.patch
file core_modules_with_last_commit.patch

nilason · 2024-02-15T14:34:55Z

Wouldn"t it be better to – in the process of creating the tarball – create and add a simple textfile containing the git commit? Configure then can try get info from git or if failing so retrieve the same info from the file (if existing).

nilason · 2024-02-15T14:49:40Z

Wouldn"t it be better to – in the process of creating the tarball – create and add a simple textfile containing the git commit? Configure then can try get info from git or if failing so retrieve the same info from the file (if existing).

Must be better than:

grass/configure.ac

Lines 157 to 176 in 25bc70d

    
           GRASS_VERSION_GIT="exported" 
        
           # get git short hash + date of last change in GRASS headers 
        
           # (and anything else in include) 
        
           GRASS_HEADERS_GIT_HASH="${GRASS_VERSION_NUMBER}" 
        
           GRASS_HEADERS_GIT_DATE=`date -u +%FT%T%z | sed "s/\(..\)$/:\1/"` 
        
           AC_PATH_PROG(GIT, git, no) 
        
           if test "$GIT" != "no" ; then 
        
              GRASS_VERSION_GIT=`$GIT rev-parse --short HEAD 2>/dev/null` 
        
              if test -z "$GRASS_VERSION_GIT"; then 
        
                 GRASS_VERSION_GIT="exported" 
        
              fi 
        
              GRASS_HEADERS_GIT_HASH=`$GIT log -1 --pretty=format:"%h" -- "${SRCDIR}/include" 2>/dev/null` 
        
              if test -z "$GRASS_HEADERS_GIT_HASH"; then 
        
                 GRASS_HEADERS_GIT_HASH="${GRASS_VERSION_NUMBER}" 
        
              fi 
        
              GRASS_HEADERS_GIT_DATE=`date -d $($GIT log -1 --pretty=format:"%cI" -- "${SRCDIR}/include") -u +%FT%T%z | sed "s/\(..\)$/:\1/"` 2>/dev/null 
        
              if test -z "$GRASS_HEADERS_GIT_DATE"; then 
        
                 GRASS_HEADERS_GIT_DATE=`date -u +%FT%T%z | sed "s/\(..\)$/:\1/"` 
        
              fi 
        
           fi

…or did I miss something?

tmszi · 2024-02-18T10:05:47Z

Wouldn"t it be better to – in the process of creating the tarball – create and add a simple textfile containing the git commit? Configure then can try get info from git or if failing so retrieve the same info from the file (if existing).

Yes it is improvement, but core_modules_with_last_commit patch/JSON file is not required file by default.

Actually all states are handled correctly during compilation process:

Git commits/date are getting from Git source code if exists
Git source code doesn"t exists (tarball) Git commits/date are getting from JSON file if exists
Git source code or JSON file doesn"t exists Git commit info will be empty and date will be time of last modification of source code path (line 218)

Info from the man page will looks like: Accessed: Sunday Feb 18 10:58:31 2024

grass/utils/mkhtml.py

Lines 216 to 221 in aaecfc8

    
           return { 
        
               "commit": "unknown", 
        
               "date": datetime.fromtimestamp(os.path.getmtime(src_dir)).strftime( 
        
                   datetime_format 
        
               ), 
        
           }

bmwiedemann · 2024-02-18T15:12:45Z

In practice, os.path.getmtime(src_dir) results in the current time being used, because we do some sed calls

nilason · 2024-02-18T15:33:28Z

Yes it is improvement, but core_modules_with_last_commit patch/JSON file is not required file by default.

The generated patch/json file is needed in a tarball release, but should be included in the tarball and used by configure. This way it results in a consistent manner.

nilason · 2024-02-18T15:43:37Z

#3435 attempt to address this for tarball releases with GRASS_HEADERS_GIT_HASH and GRASS_HEADERS_GIT_DATE, without the need for introducing new environmental variables and even further complicate the code. Next step will be to update configure and/or utils/mkhtml.py to read, if needed, from distributed json file.

nilason · 2024-02-18T16:36:00Z

Next step will be to update configure and/or utils/mkhtml.py to read, if needed, from distributed json file.

@tmszi I have not yet studied the code, but if we package the json file in the tarball, will it then automatically be used? At least that is my understanding after re-reading your comment.

tmszi · 2024-02-18T18:29:28Z

Next step will be to update configure and/or utils/mkhtml.py to read, if needed, from distributed json file.

@tmszi I have not yet studied the code, but if we package the json file in the tarball, will it then automatically be used? At least that is my understanding after re-reading your comment.

Yes it will be.

core_modules_with_last_commit.json file structure:

{
    "g.version": {
        "commit": "f146b2078dadabb79722b8e6b9323b8ae535f40f",
        "date": "2022-12-13T16:22:24-03:00"
    },
    "g.filename": {
        "commit": "547ff44e6aecfb4c9cbf6a4717fc14e521bec0be",
        "date": "2022-02-03T11:10:06+01:00"
    },
}

echoix · 2024-03-24T23:21:06Z

So now that #3435 is done, what is the status of this PR?

nilason · 2024-03-25T16:48:48Z

With #3435 and #3437, resulting in Git related data included in tarball too, consistent builds are enabled with/without Git and also across platforms. This will be available with GRASS GIS 8.4.
@bmwiedemann Thank you for bringing up this issue. Please don"t hesitate to report an issue if the fixes for 8.4 will not create reproducible builds (for Git derived data).

github-actions bot added the Python Related code is in Python label Feb 11, 2024

neteler added the libraries label Feb 11, 2024

neteler reviewed Feb 11, 2024

View reviewed changes

utils/mkhtml.py Outdated Show resolved Hide resolved

black

503b45b

github-actions bot removed the libraries label Feb 11, 2024

neteler added this to the 8.4.0 milestone Feb 11, 2024

neteler changed the title ~~Allow to override build date~~ build: Allow to override build date Feb 11, 2024

landam added general manual Documentation related issues labels Feb 12, 2024

nilason mentioned this pull request Feb 16, 2024

[Feat] Packaging: include all version data in tarball #3430

Closed

Merge branch "main" into date

7959bf5

nilason closed this Mar 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: Allow to override build date #3417

build: Allow to override build date #3417

bmwiedemann commented Feb 11, 2024

echoix commented Feb 11, 2024

tmszi commented Feb 13, 2024

tmszi commented Feb 13, 2024 •

edited

Loading

tmszi commented Feb 13, 2024

bmwiedemann commented Feb 13, 2024 •

edited

Loading

tmszi commented Feb 13, 2024

neteler commented Feb 13, 2024

tmszi commented Feb 13, 2024

tmszi commented Feb 13, 2024

tmszi commented Feb 13, 2024 •

edited

Loading

nilason commented Feb 15, 2024

nilason commented Feb 15, 2024

tmszi commented Feb 18, 2024 •

edited

Loading

bmwiedemann commented Feb 18, 2024

nilason commented Feb 18, 2024

nilason commented Feb 18, 2024

nilason commented Feb 18, 2024

tmszi commented Feb 18, 2024

echoix commented Mar 24, 2024

nilason commented Mar 25, 2024

build: Allow to override build date #3417

build: Allow to override build date #3417

Conversation

bmwiedemann commented Feb 11, 2024

echoix commented Feb 11, 2024

tmszi commented Feb 13, 2024

tmszi commented Feb 13, 2024 • edited Loading

tmszi commented Feb 13, 2024

bmwiedemann commented Feb 13, 2024 • edited Loading

tmszi commented Feb 13, 2024

neteler commented Feb 13, 2024

tmszi commented Feb 13, 2024

tmszi commented Feb 13, 2024

tmszi commented Feb 13, 2024 • edited Loading

nilason commented Feb 15, 2024

nilason commented Feb 15, 2024

tmszi commented Feb 18, 2024 • edited Loading

bmwiedemann commented Feb 18, 2024

nilason commented Feb 18, 2024

nilason commented Feb 18, 2024

nilason commented Feb 18, 2024

tmszi commented Feb 18, 2024

echoix commented Mar 24, 2024

nilason commented Mar 25, 2024

tmszi commented Feb 13, 2024 •

edited

Loading

bmwiedemann commented Feb 13, 2024 •

edited

Loading

tmszi commented Feb 13, 2024 •

edited

Loading

tmszi commented Feb 18, 2024 •

edited

Loading