catkin_make_isolated should provide rosbuild parallelity #330

wjwwood · 2013-12-03T09:25:29Z

I see no reason to not parallelize project builds of projects without dependencies between them. Code can be reused from rosmake or from rosinstall.

This might require changing the catkin_pkg API not just to return a topologically ordered list of packages, but a dependency graph (maybe without transitive dependencies)

dirk-thomas · 2013-02-13T02:04:50Z

This is not schedule to be implemented (assinged to milestone "untargeted"). Anyway patches implementing this feature are welcome.

dirk-thomas · 2013-11-26T20:10:57Z

Pass to @wjwwood.

wjwwood · 2013-12-03T09:25:41Z

Upgraded to a pull request.

wjwwood · 2013-12-03T10:16:15Z

This pull request is still a work in progress, I have chosen to rewrite the command from scratch, not reusing (by importing) anything used in the current catkin_make_isolated and catkin_make commands because of the significant restructuring of the flow control and logic. Much of the code is obviously based on or copied from the old implementations.

Some high level concepts I have been using:

Command: single action to execute as part of a group of commands which can build a package
Job: contains a list of Commands and meta data about how to build a package
- extended to typed jobs (CatkinJob and CMakeJob)
Executor: multiprocessing process which consumes Job's and executes each of the Job's Commands in order

Some things I had to address:

Synchronization of I/O: Each executor generates events, which the main thread of execution process
Capturing Color: The executors capture colored output from the subprocesses (on unix only for now)
Interleaved I/O: The output has to remain readable with proper handling for multi line and carriage return output from multiple commands simultaneously
Locking of Spaces: There is a system for commands to indicate which spaces (devel/install) it uses, so that they can be locked to prevent multiple jobs from writing the same file at the same time.
- Currently only for the install space
- May not be necessary

Currently I can build almost the entire hydro-desktop-full workspace on my mac, and the remaining build errors do not appear to be related to this command (they fail in catkin_make_isolated as well right now). I already found three instances of missing calls to include_directories in various packages using this command. I have mostly been doing unmerged devel space and no install.

This is what remains to be done from my point of view:

These are some additional TODO's, which would be nice to have, but not strictly necessary:

Add checks for reusing build/devel spaces with different command options, i.e. warn when using the same devel space for merged and nonmerged devel on seprarte runs of the command
Automated tests for all of the corner cases

wjwwood · 2013-12-20T06:10:11Z

@jonbinney d7b2b26

wjwwood · 2013-12-26T18:33:26Z

Ok, 86a89b2 seems to work for me too.

wjwwood · 2013-12-27T02:19:58Z

@dirk-thomas can you look at this commit 0548d50?

Does that make sense as a better default?

dirk-thomas · 2013-12-27T18:02:06Z

Yeah, that looks much more straight forward.

wjwwood · 2013-12-27T21:32:17Z

Ok, I address the comments and I removed the last os.remove as the file should not exist after os.rename anyways.

this single implementation in the catkin.cmi.common module also can cache the result because calculating it is expensive and done each time a job finishes

you can now: - build explicit packages in the workspace - choose to not build deps with --no-deps - passing arbitrary cmake/make args now works - only list the packages to be built, do not build These options were add but are not implemented: - skip packages up to a certain pkg, --start-with - order the output of commands into blocks

pcmi now outputs delayed, ordered output by default, and you can instruct it to output interleaved data without a delay. It will also printout data without delay when only one executor is being used.

this might not be needed

This also removes duplicates

…ible

This should allow better capture of `\r`'s

wjwwood · 2014-03-12T01:41:05Z

I am going to close this pull request in favor of the implementation in the new catkin_tools package as catkin build:

https://github.com/catkin/catkin_tools

I'll leave the branch around for people who might be using it.

jbohren · 2014-03-12T01:48:47Z

Thanks for the update, Will. First glance at the docs and this is looking pretty awesome!

wjwwood · 2014-03-12T01:49:38Z

Sure, I'm hoping to get this into a good state for Indigo.

jbohren · 2014-03-12T01:54:27Z

I just downloaded and installed it from source. I ran catkin build and said "woah" out loud.

wjwwood · 2014-03-12T01:55:20Z

Colors man, colors.

tkruse · 2014-03-12T08:03:53Z

well done

dirk-thomas mentioned this pull request Nov 18, 2013

catkin_make_isolated not isolated #367

Closed

ghost assigned wjwwood Nov 26, 2013

wjwwood mentioned this pull request Dec 7, 2013

linkedit error, argument list too long #558

Closed

wjwwood mentioned this pull request Dec 21, 2013

added atomic_configure_file function to catkin #566

Merged

wjwwood mentioned this pull request Jan 16, 2014

catkin_make --pkg includes CMakeLists from all packages in workspace, not just those needed to build the requested packages ros-infrastructure/catkin_pkg#71

Closed

wjwwood added 16 commits January 16, 2014 18:07

initial commit of the parallel version of cmi

a48fd6c

Fix issues with absolute paths

00ee31a

implement getting recursive build deps in one func

0fc64ff

this single implementation in the catkin.cmi.common module also can cache the result because calculating it is expensive and done each time a job finishes

remove vestigial function

d5f1cef

pcmi: output ordering and changed default behavior

614efa1

pcmi now outputs delayed, ordered output by default, and you can instruct it to output interleaved data without a delay. It will also printout data without delay when only one executor is being used.

remove locking for devel and install spaces

1272675

improve output of pcmi

e5fb428

added --space-suffix option

12f329d

fixes for --install

2b476bf

try and capture all data from subprocesses

3f341ac

this might not be needed

write job logs to file

2f592e5

implemented --start-with

6deee07

do not use --extend when only one source

9cd3e54

strip ansi escape characters in output files

165e672

sort packages after determining what will be built

6927361

This also removes duplicates

wjwwood and others added 21 commits January 16, 2014 18:07

remove escape sequence printed with color disabled

e640ebe

fix disabling of wide_log

9878efd

fix disable of colorization on some outputs

cd9f8dd

Add pcmi to the setup.py scripts list

7e56f64

Update the relative import rule in the pcmi script

2cffdef

fix install job generation

33d11d2

fix py3 import of Empty from (Q)queue

756b37d

remove dead code

fdea695

fix some corner cases in the build setup and loop

87eed96

improved API documentation

f1fefa5

added total run time to status bar

c0c6f8a

fix for cmi_env.sh sourcing of setup.sh on linux

5d1fae7

improved fix to sourcing in cmi_env.sh

de0a0a5

pcmi: add special handling when rename between partitions is not poss…

882b7f6

…ible

change the install lock to be on by default

99110ec

improve printing performance of command output

b22b3dc

fix passing of catkin_make_args to make for catkin packages

09da106

improved atomic creation of setup files

7646d2c

touch up to setup file generation

b8218b5

preserve line endings further in the output chain

5524c9d

This should allow better capture of `\r`'s

fix bug which caused repeated lines in the output

66d8942

wjwwood closed this Mar 12, 2014

wjwwood mentioned this pull request Mar 13, 2014

catkin build feedback catkin/catkin_tools#7

Closed

3 tasks

dirk-thomas deleted the parallel_cmi branch November 11, 2019 19:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

catkin_make_isolated should provide rosbuild parallelity #330

catkin_make_isolated should provide rosbuild parallelity #330

wjwwood commented Dec 3, 2013

dirk-thomas commented Feb 13, 2013

dirk-thomas commented Nov 26, 2013

wjwwood commented Dec 3, 2013

wjwwood commented Dec 3, 2013

wjwwood commented Dec 20, 2013

wjwwood commented Dec 26, 2013

wjwwood commented Dec 27, 2013

dirk-thomas commented Dec 27, 2013

wjwwood commented Dec 27, 2013

wjwwood commented Mar 12, 2014

jbohren commented Mar 12, 2014

wjwwood commented Mar 12, 2014

jbohren commented Mar 12, 2014

wjwwood commented Mar 12, 2014

tkruse commented Mar 12, 2014

catkin_make_isolated should provide rosbuild parallelity #330

catkin_make_isolated should provide rosbuild parallelity #330

Conversation

wjwwood commented Dec 3, 2013

dirk-thomas commented Feb 13, 2013

dirk-thomas commented Nov 26, 2013

wjwwood commented Dec 3, 2013

wjwwood commented Dec 3, 2013

wjwwood commented Dec 20, 2013

wjwwood commented Dec 26, 2013

wjwwood commented Dec 27, 2013

dirk-thomas commented Dec 27, 2013

wjwwood commented Dec 27, 2013

wjwwood commented Mar 12, 2014

jbohren commented Mar 12, 2014

wjwwood commented Mar 12, 2014

jbohren commented Mar 12, 2014

wjwwood commented Mar 12, 2014

tkruse commented Mar 12, 2014