Note this discussion started in another #393 but is now migrated to here.

@jedbrown (that is not completing so I am not sure how Jed was on the earlier CGNS ticket but not here) contributed this picture in CNDA Slack and in the prior CGNS issue #296 that has been closed. From cgnsview

With the following description (that I will lace my questions into as I try to collect the data to throw at these nodes, I am looking for a bit more detail about what is needed in each node/leaf).

• One contiguous GridCoordinates, which gives a unique global ID to each node clear

• TetElements followed by PyramidElements followed by PrismElements cover the volume. There are two entries here. ElementRange and ElementConnectivity. The former seems to need to be continuous, starting from from 1 for a given zone and continue as you move to a new topology of a given zone. Obviously we can can create this numbering but I don't think SCOREC/core will spit them out of a single iterator like this but with multiple iterators and a conditional and counter it should not be hard. As I look at the documentation, it shows ElementType_t between ElementRange and ElementConnectivity. Does cgnsview just not show this or, it it promoting that to the foldername? I think no because it carries number of nodes (e.g., Hexa_27). ElementConnectivity is a table (all nodes of an element per row).

This example has 3 element groups in a single zone: PyramidElements, TetElements, and Prism Elements

What FSFBrackets are right after PrismElements and before ZoneBC? Is this just a boundary element mesh set that got written before before the ZoneBC after which the rest? Was this by design or a glitch to the organization?
ZoneBC is next but the description skips to...

• Each semantic surface (like Nacelle here) has a Elements_t node at the same level (under blk-1) with the element connectivity. This is a Pointwise convention, and it would also be possible to lump all the triangle faces into a single file node. If I understand this correctly, these are a single element group for each model face following the same convention as the volume elements above. Specifically, I think the first set will range from 1 to the number of mesh faces on the first model face, followed by ElementType, followed by the connectivity table.

• Under ZoneBC, there is a BC_t node for each semantic surface, pointing to the element range (which is contiguous because of the Pointwise convention, but need not be) Do we need these or are we going to derive all boundary conditions from model face mesh sets in the prior paragraph?

• For each BC, there is also a Family_t node under Base that can offer more semantic information about that boundary type. The screenshot here doesn't use that in a meaningful way. Same question?

• Note that Symmetry contains both triangles and quads, so there are two Elements_t nodes in the file, tri_Symmetry and quad_Symmetry, with element numbering contiguous across the two. The corresponding BC_t Symmetry node refers to the contiguous range over both face topologies. Clear.

• Note that you generally don't manage writing these file nodes manually, but rather through the Boundary Condition interfaces. @jrwrigh we should find a time to discuss this. I am currently reviewing what chef does to find the places we can extract this data from the mds database but I assume to use these functions we will have to put it into the specific data structures that these interfaces require. If we get that right your work might be pretty easy.

• To answer your question, yes this is more of the "mesh sets" model. In PETSc, we'll create the volume elements and interpolate the mesh (create the faces and edges), then broker identification of the faces through a vertex (to keep communication structured and scalable) so we can add them to the appropriate Face Sets.
  OK my largest question is whether we will or will not need ZoneBC information that I have skipped above (thinking we don't need it). Second question is how we plan to handle periodicity. I think Jed discussed that somewhere so I will find that and surface that in another comment.

Notes: (see CPEX 0031)
The use of ElementList and ElementRange for ptset_type is deprecated and should not be used in new code for writing boundary conditions. These are still currently accepted, but will be internally replaced with the appropriate values of PointList/PointRange and GridLocation_t, based on the base cell dimension.
Code which reads older CGNS files, should handle ElementList and ElementRange, however, since many older files contain these specifications for ptset_type.
Is from the link @jedbrown gave to Boundary Condition interfaces. It looks like they don't want boundary conditions to be given in terms of face sets which is unfortunate in my opinion. Boundary conditions are attributes of model faces. Mesh "nodes" on model edges and model vertices are best resolved by inheritance rules from the intersection of faces. Obviously we can do that work on the SCOREC core side and produce node lists for each boundary condition but I am not clear yet whether PETSc wants these node lists or wants to do those inheritance rules and intersection from faces lists.

Weak BCs certainly belong on faces. Are they suggesting that these also should be given as node lists?

Jed said:

Regarding BCs, GridLocation_t can take the value FaceCenter, so this is just a naming thing (preferring to call it PointList with FaceCenter instead of ElementList). Notable that Pointwise is still creating files with the old convention.

Jed said (much more)

As I look at the documentation, it shows ElementType_t between ElementRange and ElementConnectivity. Does cgnsview just not show this or, it it promoting that to the foldername?

You'll see it if you click on the "folder" nodes. The 10 appearing on the right is the enum value from the source code

typedef enum {
  CGNS_ENUMV( ElementTypeNull  ) =CG_Null,
  CGNS_ENUMV( ElementTypeUserDefined ) =CG_UserDefined,
  CGNS_ENUMV( NODE ) =2,
  CGNS_ENUMV( BAR_2 ) =3,
  CGNS_ENUMV( BAR_3 ) =4,
  CGNS_ENUMV( TRI_3 ) =5,
  CGNS_ENUMV( TRI_6 ) =6,
  CGNS_ENUMV( QUAD_4 ) =7,
  CGNS_ENUMV( QUAD_8 ) =8,
  CGNS_ENUMV( QUAD_9 ) =9,
  CGNS_ENUMV( TETRA_4 ) =10,
  CGNS_ENUMV( TETRA_10 ) =11,
  CGNS_ENUMV( PYRA_5 ) =12,
  CGNS_ENUMV( PYRA_14 ) =13,
  CGNS_ENUMV( PENTA_6 ) =14,
  CGNS_ENUMV( PENTA_15 ) =15,
  CGNS_ENUMV( PENTA_18 ) =16,
  CGNS_ENUMV( HEXA_8 ) =17,
[...]

What FSFBrackets are right after PrismElements and before ZoneBC?

The ordering in cgnsview is arbitrary from what I can tell. It's just a surface with a few hundred triangles, and has a corresponding entry in ZoneBC.

Each semantic surface (like Nacelle here) has a Elements_t node at the same level (under blk-1) with the element connectivity. This is a Pointwise convention, and it would also be possible to lump all the triangle faces into a single file node. If I understand this correctly, these are a single element group for each model face following the same convention as the volume elements above. Specifically, I think the first set will range from 1 to the number of mesh faces on the first model face, followed by ElementType, followed by the connectivity table.

Mostly, though usually one writes the volume elements first so those start at 1 and the face numbers start after that. If you click around in this file, the element blocks are numbered sequentially with tet, pyramid, prism, then FSFBrackets (triangles), etc. The ordering here is arbitrary. Note that usually the mapping to file schema is handled by the library (we won't be reading/writing raw HDF5, but rather calling the CGNS interfaces that create these linked nodes). But I've certainly found it useful to explore using cgnsview to see how a file was intended to be used.

Under ZoneBC, there is a BC_t node for each semantic surface, pointing to the element range (which is contiguous because of the Pointwise convention, but need not be) Do we need these or are we going to derive all boundary conditions from model face mesh sets in the prior paragraph?

They're supposed to be present and will be written when using the API. Here's an example from the test suite (includes multi-zone unstructured)

For each BC, there is also a Family_t node under Base that can offer more semantic information about that boundary type. The screenshot here doesn't use that in a meaningful way. Same question?

This appears optional, and is not written in the unstructured test. I haven't seen an example in which it's used for something I can recognize as important.

OK my largest question is whether we will or will not need ZoneBC information that I have skipped above (thinking we don't need it).

We want that because it's the way BCs are identified. It is automatically written by the interface. If for some reason we need naming heuristics instead, we could handle that, but we may as well write compliant files.

Second question is how we plan to handle periodicity. I think Jed discussed that somewhere so I will find that and surface that in another comment.

There is cg_conn_write. The docs (excerpt quoted below) describes how one can either match vertices to donor vertices or faces to donor faces. Note that CGNS vertices are all the nodes while PETSc vertices are just corners (of arbitrary order parametric elements). So my preference is to describe periodicity in terms of faces, which can be directly fed into the isoperiodic support. Life will be easiest if those face elements are oriented the same way on the periodic and donor surfaces.

For Abutting or Abutting1to1 interfaces, GridLocation can be either Vertex or FaceCenter. When GridLocation is set to Vertex, then PointList or PointRange refer to node indices, for both structured and unstructured grids. When GridLocation is set to FaceCenter, then PointList or PointRange refer to face elements. Face elements are indexed using different methods depending if the zone is structured or unstructured.

Thanks @jedbrown this is much more clear now. Thanks also to @jrwrigh who explained quite a bit of this to me when we met on Tuesday to discuss this. As you can see @cwsmith , I have made PR out of my WIP branch. Clearly all the real development described in the PR description still needs to be "done" but I hope you don't mind that we use the PR review utility as a communication platform to plan, accelerate, and document our progress. @cwsmith, I think I have copied everything from the prior ticket so if you want to delete those comments to keep the PR short, sweet, and focused just on the subject (relaxing/removing the requirement for C 14 in the CGNS development that then exposed SCOREC/core code that was not C 14 compliant). I think I have merged those changes into this branch which does build a chef executable with CGNS enabled (which of course is necessary for this PR's development).

DEVELOPMENT ROADMAP

1. Fix SCOREC-core’s CGNS writer to output CGNS data structures that have boundary element connectivity with the same node numbering used on the interior. The current SCOREC-core capability writes its example from “withBCS” in a way that we cannot use so the sub steps of this task are: a) KEJ figures out how to get the data we need in one “base”, b) JRW figures out how to use intermediate CGNS functions to write a CGNS file in a way that PETSc can read and setup CEED-PHASTA problems.
2. The above could allow us to make some initial tests to verify pieces are working e.g., a) all hex case that can be verified against the PETSc box mesh approach, b) all tet case, c) all wedge case which will already need the writer to distinguish between quad and tri elements and will require libCEED to be multi-topology aware at least on faces….James mentioned there is a test somewhere already for a mix of quads and tris that perhaps we can draw on.
3. Even before 2c we could attempt to do an all hex or all tet case of the bump with existing libCEED working from a serial, SCOREC-core written mesh from step 1. above. This also could be tested on our flat plate 12-30 and 6-15 cases. However, this is likely to be only a proof of concept if we have to go through a serial mesh ingestion for each run thus….
   Parallel CGNS which I think Jed has volunteered for.

Note, I am not super clear on point 3. @jedbrown How does CEED-PHASTA run continuation work? Are files geometry and boundary condition files written at the end of a run that allow continuation of that run with the exiting partition or, are we currently pulling the solution back to a serial CGNS file for visualization and thus needing work (and a plan) to save more data (in parallel?) to be used as a checkpoint that avoids redoing the partition and boundary condition creation?

On the last point, it will be possible to read any cgns checkpoint as mesh and initial condition. If the number of ranks matches, we can continue with the same partition, otherwise we'll repartition and continue.

Life will be easiest if those face elements are oriented the same way on the periodic and donor surfaces.
I see three issues here to clarify completely what oriented the same way means:

1. I assume we want one face element list for what you call periodic and a separate face element list for what you call donor (I say you call just because we called them other things at various times)
2. I assume oriented at least means that they appear as a matched list (ith element in the periodic list covers the same area, bounded by the same edges, bounded by the same nodes as the ith element in the donor list).
3. Finally the question, by oriented the same way do you mean you want node ordering (and thus inferred boundary mesh ordering to MATCH) which would in fact make the normals created by any two adjacent edges point in the same direction and thus one points into the domain and one points out of the domain.

@jedbrown, I ask this question because, historically, for PHASTA, we have always created volume elements for the boundary elements with the first n_face_verts ordered as they should be for that volume element according to PHASTA's convention (and shape function definition). This has been complicated (and there are appropriately brutal complaints in the comments for the fact) by tet elements being negative volume oriented (curling the tri face always points away from the opposing vertex) while all others follow the typical FE convention of positive elements (curl of first two edges points to opposing face).

I guess this whole comment can be TLD'R to the following questions for @jedbrown:

1. Do we want our boundary elements to have connectivity ordering to point into or out of our domain?
2. Shall we maintain that convention (inward or outward ordering for all) for periodic and donor face and just use the same STARTING vertex which will require downstream code to recognize that node matches will be found in the opposite order of the node list for an element pair OR
   shall we make the node lists match and make a choice that the face called periodic has an outward normal while the one labeled donor has an inward facing normal (or vice versa but we do need to make an explicit choice I think).
3. For multiple direction periodicity, I assume will just give an index to each donor-periodic face list pair (e.g., donor1 matches periodic1, donor2 matches periodic2, and so on) and we will leave it to PETSc to work out that two directions of periodicity will require mesh edges at intersections of the two periodic/donor pairs to become 3 donors to a single edge and further that triple periodicity does this plus 7 donors for a single periodic vertex.

On the last point, it will be possible to read any cgns checkpoint as mesh and initial condition. If the number of ranks matches, we can continue with the same partition, otherwise we'll repartition and continue.

Just to be possibly pedantically clear, does mesh in this phrase include boundary condition information (the boundary element sets and ZoneBC) with some concept of the partition used in the simulation that wrote the checkpoint such that, when continuing at the same part count, it is mostly reload and go (without redoing rendevous work that I assume will have to be done the first time PETSc loads the serial or parallel CGNS mesh to start the simulation)?

After meeting with @cwsmith and @jrwrigh this morning, and then thinking a bit about it during my bike ride in to campus it occurs to me that we do have another route that we could consider. I had dismissed it earlier for a variety of reasons but I guess I want to air it out here for support/criticism.

The idea is to write let SCOREC-core/chef stay limited to writing posix PHASTA files (which means one file per part) and write a code that reads those files and writes a CGNS file. The upside is that all the CGNS facing code could be fortran based (allowing us to follow their examples exactly and not map all of it to C which I think @jrwrigh and I are less than strong in).

The downside is that I am less than sure of what CGNS wants us to "write" into a file to express parallel.

This is also what is paralyzing my decision to update the CGNS writer that is already in SCOREC-core since it hard for me to design it without a vision of how it gets written in parallel. Maybe I need to say a little more about how the data is laid out for @jedbrown to help me clear that mental/writers block.

After partitioning the MDS database, there is a distinct set of elements on each part and they retain classification against a real geometric model (if it exists) or a topological model if it does not. What chef the pre-processor for PHASTA does is to have each part create a connectivity for the volume elements on each part and a distinct connectivity for the boundary elements on each part of each topology. This last part is a little subtle in that PHASTA wants volume elements on the boundary so this is actually a block of elements for each volume-element-face-type topology on this process/part (yes chef is limited to run on the same number of processes as their are parts). If I have an all-tet mesh there will only be tri-faces on the boundary but if my part has elements on two or more model faces (e.g., if in a corner) all those tets will be in one element connectivity. This is easily reconciled to sort to particular boundaries because we can give each model face a surfID attribute and each of these elements will then have this array to sort it to what CGNS wants. For complete clarity, if a volume element has faces on 2 or 3 boundaries it will appear 2 or 3 times respectively each time with the ordering permuted such that the first 3 nodes are on the model face. All wedge meshes are handled similarly with tris bumping to quads. All wedge meshes bring the added complexity of having quads on some faces (first 4 of the 6 nodes permuted to be on the boundary) and some quad faces but, in most usages a face will be all quad or all tri so not that complex. Mixed meshes could have multiple blocks of elements even of the same face shape on a given model face but chef would put these into different blocks (e.g., tris from tets would be in a different block than tris from wedges or pyramids; quads from hexes would be in a different block from quads from wedges or pyramids). I mention pyramids but in truth, a good mesh seldom has pyramids that have a face on the boundary.

@cwsmith pointed out that it is not so simple to have O(10k) processes write their "chunk" of:

1. coordinates (we do have a concept of owned and non-owned vertices so at the very least we have to be careful to only write the owned AND I guess we need to think about what CGNS and later PETSc want to see which is a linear range of owned vertices in a global numbering aligned with rank)
2. volume connectivity (not in the on-part node numbering but in the global numbering described in 1.)
3. for each boundary, for each face topology, a surface connectivity like 2. with numbering increasing linear/continuous from the end of the prior connectivity (which starts from the end number of volume connectivity for the first boundary element group). I guess this is not strictly required as we could write one connectivity per face topology and just create a same sized integer array that tells us which boundary each face is on (which chef already produces for PHASTA). This might be a lot simpler give the complexity described above in terms of combinations (which were made more complicated by PHASTA requiring volume elements for the boundary).

So, at long last my question: given the above data layout on say 10-100k ranks, can the provided CGNS library file writers work with the data distributed among ranks like this (assuming we have filled the arrays we pass to it with the correct global numbers for that rank's data?

Parsing that question into each category:

1. if coordinates are pruned down to the owned nodes (no replicas from other ranks) will call cg_coord_write_f(index_file,index_base,index_zone,RealDouble, 'CoordinateX',x,index_coord,ier) on each of the 10k ranks result in a correct coordinates file to be loaded on 10k ranks of PETSc?
2. if connectivity arrays are mapped from their "current" on-rank numbering the numbering of the coordinates written in 1. will call cg_section_write_f(index_file,index_base,index_zone, 'Elem',HEXA_8,nelem_start,nelem_end,nbdyelem,ielem, index_section,ier) on 10k ranks result in a correct connectivity file to be loaded on 10k ranks of PETSc.
3. similar for boundary, similar for ZoneBC but boundary elements instead of volume.

OR
is it on us to do the parallel write.

I guess that is my real question. Are those CGNS writers going to handle the parallel writing leaving our job to be only getting the data from rank-local numbering to the linearly increasing numbering?
If those calls to cgns "handle" the parallel writing for us, I already have a lot of that local to global code written in the version of PHASTA that uses PETSc (because PETSc requires it) which means that I might get this done faster by writing a fortran code that:

1. on N ranks reads the N posix files chef wrote
2. computes the node number mapping
3. compacts the arrays to the size the CGNS library writer functions want (with that we passed to them appropriately mapped to global numbering)
4. calling the CGNS libraries to write the files.

Obviously a strong C coder could do the same within chef but I am not sure we have one of those. Further, obviously the serial version of this code is pretty simple AND follows the examples given here.

I'm not sure if I'm reading your whole post correctly, but this is how we do it in PETSc. Each rank creates a connectivity array for the elements it owns, and the collective write is done in these two lines (the first is metadata, the second is array data). https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/cgns/plexcgns2.c?ref_type=heads#L791-792

The vertex numbers are with respect to the node numbering used for GridCoordinates, which use a similar two-stage write here.

@jedbrown thanks for the links to how PETSc sets up these files and writes data. For now I have abandoned the clunky idea of making a post-chef converter from posix to CGNS and returned to this branch to attempt to get these files written directly from Chef so we (@jrwrigh and I) should be able to mimic the PETSc writing closely. In the commit directly above, I believe I have the functions added to get the data for region connectivity and a surface connectivity in the form needed. Further, I think the coordinates are collected into an array that matches the expectations of CGNS (all x, then all y then all z though I am not sure why somone put a comment saying FORTRAN indexing??? I suppose they mean to say flat array that runs over points with the first linear index first and the dim of point as the second index not 1 bases as we or at least I usually think of as Fortran indexing. C centric folks don't always appreciate the value of flat arrays over arrays of arrays.)

static void getCoordinates(Output& o)
{
  apf::Mesh* m = o.mesh;
  int n = m->count(0);
  double* x = new double[n * 3];
  apf::MeshEntity* v;
  int i = 0; 
  apf::MeshIterator* it = m->begin(0);
  while ((v = m->iterate(it))) {
    apf::Vector3 p;
    m->getPoint(v, 0, p);
    for (int j = 0; j < 3;   j) 
      x[j * n   i] = p[j]; /* FORTRAN indexing */
      i; 
  }
  m->end(it);
  PCU_ALWAYS_ASSERT(i == n);
  o.arrays.coordinates = x; 
}

so it should be simple (serial for now) to throw three continuous slices of this array at the CGNS writer. The o data structure is passed to the retained but not updated writer function at the bottom of phCGNSgbc.cc though I guess if we don't want to work with that we could just inline this code and pass ranges of x to CGNS. That might be the best approach anyway since we are going to have to compact and map this numbering for parallel anyway.

@jrwrigh do you want to take a cut at the CGNS file write stuff (set up for parallel since that is surely the way we will go) that so that we can get a serial test of coordinates volume connectivity and then add the boundary element lists? That would free me to work on creating the map from on-rank numbering to global numbering so that we could then follow up that test with a parallel test soon? I am available to help when if that is needed.

I think I have all the data in flat 1-D arrays for coordinates, volume connectivity, and boundary element connectivity to be chucked into a CGNS file in a parallel way. I am pretty terrible at function interfaces so I pretty much pushed everything that needs to be shared into the output data structure (added stuff in phasta/phOutput.h ). I lifted the complicated code from a PHASTA routine to find the mapping of PHASTA on rank numbering to PETSc global numbering based on on-rank-owned number sum of lower ranks owned number (linear global numbering with rank) which I interpreted CGNS to want as well. That is by far the most complicated part of this development and thus, assuming I did not introduce bugs as I changed its interface (which were the only intended changes) I am hopeful this does not have too many bugs.

Work left to do:

1. Add the code to do the CGNS writes
2. Add the code/flags to tell chef we want CGNS writes
3. At that point we can create some test geometries and look at results in PV.
4. I will set up those test cases with surfIDs numbers that match our 1-6 so that the next step can be to pluck out the boundary element numbers for each "type" of boundary condition. I am thinking that ZonalBC coding will be easy if I provide an array of the same size as the number of boundary elements to make that relatively painless.

I will likely plug away at 2-4.

As I start to add the calls to CGNS to write the CGNS file, I am noticing some places to easily get confused:

1. chef was already preparing for PHASTA files writeInts(f, " mode number map from partition to global", o.arrays.globalNodeNumbers, m->count(0));. This is a map back to the ORIGINAL global numbering and not what CGNS/PETSc want (linearly increasing in rank ownership). We should consider a chef flag to choose one or the other (or both) to write into PHASTA files to save doing this work in PHASTA at the start of every run that uses the PETSc solver.
2. I am deleting the code that retrieves data to write into the PHASTA posix file headers but, if we find we again need some numbers for CGNS examples of retrieval can be located in phGeomBC.cc
3. while the link @jedbrown provided above provides a pretty clear map to how the data is written, to see how PETSc opens the file I am reading from here petsc/src/sys/classes/viewer/impls/cgns/cgnsv.c
4. My struggles with interfaces continue. Seems a straight copy out of the example CGNS publishes is also not loved by evolving C standards but I guess it is just a warning. I will rely on @cwsmith to make this ugly first step better

[ 89%] Building CXX object phasta/CMakeFiles/ph.dir/phCGNSgbc.cc.o
/projects/tools/SCOREC-core/core/phasta/phCGNSgbc.cc: In function 'void ph::writeCGNS(ph::Output&, std::string)':
/projects/tools/SCOREC-core/core/phasta/phCGNSgbc.cc:311:26: warning: ISO C   forbids converting a string constant to 'char*' [-Wwrite-strings]
  311 |   static char *outfile = "chefOut.cgns";

@cwsmith

13 tests added and they pass. A couple of old ones fail. I only built chef. Do these tests need more stuff built?

(base) kjansen@viz003: /projects/tools/SCOREC-core/buildCGNSandTest $ ctest -R chef
Test project /projects/tools/SCOREC-core/buildCGNSandTest
      Start  27: chef-CGNS-multitopology1
 1/42 Test  #27: chef-CGNS-multitopology1 .........   Passed    3.27 sec
      Start  28: chef-CGNS-multitopology1-diff
 2/42 Test  #28: chef-CGNS-multitopology1-diff ....   Passed    0.15 sec
      Start  29: chef-CGNS-multitopology2
 3/42 Test  #29: chef-CGNS-multitopology2 .........   Passed    3.77 sec
      Start  30: chef-CGNS-multitopology2-diff
 4/42 Test  #30: chef-CGNS-multitopology2-diff ....   Passed    0.13 sec
      Start  31: chef-CGNS-multitopology4
 5/42 Test  #31: chef-CGNS-multitopology4 .........   Passed    3.63 sec
      Start  32: chef-CGNS-multitopology4-diff
 6/42 Test  #32: chef-CGNS-multitopology4-diff ....   Passed    0.14 sec
      Start  33: chef-CGNS-8hex1
 7/42 Test  #33: chef-CGNS-8hex1 ..................   Passed    0.27 sec
      Start  34: chef-CGNS-8hex1-diff
 8/42 Test  #34: chef-CGNS-8hex1-diff .............   Passed    0.03 sec
      Start  35: chef-CGNS-8hex2
 9/42 Test  #35: chef-CGNS-8hex2 ..................   Passed    0.25 sec
      Start  36: chef-CGNS-8hex2-diff
10/42 Test  #36: chef-CGNS-8hex2-diff .............   Passed    0.03 sec
      Start  37: chef-CGNS-smallTet1
11/42 Test  #37: chef-CGNS-smallTet1 ..............   Passed    0.25 sec
      Start  38: chef-CGNS-smallTet1-diff
12/42 Test  #38: chef-CGNS-smallTet1-diff .........   Passed    0.04 sec
      Start  39: chef-CGNS-smallTet2
13/42 Test  #39: chef-CGNS-smallTet2 ..............   Passed    0.32 sec
      Start  40: chef-CGNS-smallTet2-diff
14/42 Test  #40: chef-CGNS-smallTet2-diff .........   Passed    0.04 sec
      Start  41: chef-CGNS-AllHex1
15/42 Test  #41: chef-CGNS-AllHex1 ................   Passed    0.27 sec
      Start  42: chef-CGNS-AllHex1-diff
16/42 Test  #42: chef-CGNS-AllHex1-diff ...........   Passed    0.03 sec
      Start  43: chef-CGNS-AllHex2
17/42 Test  #43: chef-CGNS-AllHex2 ................   Passed    0.30 sec
      Start  44: chef-CGNS-AllHex2-diff
18/42 Test  #44: chef-CGNS-AllHex2-diff ...........   Passed    0.03 sec
      Start  45: chef-CGNS-AllTet
19/42 Test  #45: chef-CGNS-AllTet .................   Passed    0.33 sec
      Start  46: chef-CGNS-AllTet-diff
20/42 Test  #46: chef-CGNS-AllTet-diff ............   Passed    0.03 sec
      Start  47: chef-CGNS-AllTet2
21/42 Test  #47: chef-CGNS-AllTet2 ................   Passed    0.34 sec
      Start  48: chef-CGNS-AllTet2-diff
22/42 Test  #48: chef-CGNS-AllTet2-diff ...........   Passed    0.02 sec
      Start  49: chef-CGNS-AllWedge1
23/42 Test  #49: chef-CGNS-AllWedge1 ..............   Passed    0.30 sec
      Start  50: chef-CGNS-AllWedge1-diff
24/42 Test  #50: chef-CGNS-AllWedge1-diff .........   Passed    0.03 sec
      Start  51: chef-CGNS-AllWedge2
25/42 Test  #51: chef-CGNS-AllWedge2 ..............   Passed    0.30 sec
      Start  52: chef-CGNS-AllWedge2-diff
26/42 Test  #52: chef-CGNS-AllWedge2-diff .........   Passed    0.03 sec
      Start  53: chef-BL_query
27/42 Test  #53: chef-BL_query ....................   Passed    0.28 sec
      Start  54: chef-BL_query-diff
28/42 Test  #54: chef-BL_query-diff ...............   Passed    0.01 sec
      Start 174: chefStream
29/42 Test #174: chefStream .......................***Failed    0.08 sec
      Start 175: chef0
30/42 Test #175: chef0 ............................   Passed    4.14 sec
      Start 176: chef1
31/42 Test #176: chef1 ............................   Passed    0.01 sec
      Start 177: chef2
32/42 Test #177: chef2 ............................   Passed    0.01 sec
      Start 178: chef3
33/42 Test #178: chef3 ............................   Passed    5.19 sec
      Start 179: chef4
34/42 Test #179: chef4 ............................   Passed    2.02 sec
      Start 180: chef5
35/42 Test #180: chef5 ............................   Passed    4.54 sec
      Start 181: chef6
36/42 Test #181: chef6 ............................   Passed    1.64 sec
      Start 182: chef7
37/42 Test #182: chef7 ............................   Passed    0.01 sec
      Start 183: chef8
38/42 Test #183: chef8 ............................   Passed    0.01 sec
      Start 184: chef9
39/42 Test #184: chef9 ............................   Passed    3.66 sec
      Start 185: chefReadUrPrep
40/42 Test #185: chefReadUrPrep ...................***Failed    0.07 sec
      Start 186: chefReadRibUrPrep
41/42 Test #186: chefReadRibUrPrep ................***Failed    0.06 sec
      Start 187: chef10
42/42 Test #187: chef10 ...........................   Passed    3.48 sec

93% tests passed, 3 tests failed out of 42

Total Test time (real) =  39.62 sec

The following tests FAILED:
	174 - chefStream (Failed)
	185 - chefReadUrPrep (Failed)
	186 - chefReadRibUrPrep (Failed)
Errors while running CTest

Note the new tests use cgnsdiff. Regular diff always fails (presumably because there are file creation dates or other within the file thatcgnsdiff "knows" to ignore). @jrwrigh and I theorize that if your terminal has the means to properly build and link to cgns (e.g. spack loaded) then cgnsdiff would be in path.

Thank you for adding the tests.

cgnsdiff

We should use find_program(...) to check if cgnsdiff exists and then enable the <testname>-diff tests if it was found.

failures

I'm not sure about those. Would you please paste the log/output (ctest -V --rerun-failed)?

my bad. These are cases I have not built. The chef wildcard picked them up.

(base) kjansen@viz003: /projects/tools/SCOREC-core/buildCGNS_OneBase $ ctest  -V --rerun-failed -R chef 
UpdateCTestConfiguration  from :/projects/tools/SCOREC-core/buildCGNS_OneBase/DartConfiguration.tcl
UpdateCTestConfiguration  from :/projects/tools/SCOREC-core/buildCGNS_OneBase/DartConfiguration.tcl
Test project /projects/tools/SCOREC-core/buildCGNS_OneBase
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 174
    Start 174: chefStream

174: Test command: /usr/local/spack/v0.16/opt/spack/linux-debian8-nehalem/gcc-11.2.0/openmpi-4.1.1-c6dl75o4qks7thuaqfe4d4acbpugbiit/bin/mpirun "-np" "1" "/projects/tools/SCOREC-core/buildCGNS_OneBase/test/chefStream"
174: Test timeout computed to be: 10000000
174: --------------------------------------------------------------------------
174: mpirun was unable to launch the specified application as it could not access
174: or execute an executable:
174: 
174: Executable: /projects/tools/SCOREC-core/buildCGNS_OneBase/test/chefStream
174: Node: viz003
174: 
174: while attempting to start process rank 0.
174: --------------------------------------------------------------------------
1/3 Test #174: chefStream .......................***Failed    0.08 sec
test 185
    Start 185: chefReadUrPrep

185: Test command: /usr/local/spack/v0.16/opt/spack/linux-debian8-nehalem/gcc-11.2.0/openmpi-4.1.1-c6dl75o4qks7thuaqfe4d4acbpugbiit/bin/mpirun "-np" "4" "/projects/tools/SCOREC-core/buildCGNS_OneBase/test/chefReadUrPrep" "../../../model.dmg" "bz2:../good_mesh/" "adapt.ur.inp"
185: Test timeout computed to be: 10000000
185: --------------------------------------------------------------------------
185: mpirun was unable to launch the specified application as it could not access
185: or execute an executable:
185: 
185: Executable: /projects/tools/SCOREC-core/buildCGNS_OneBase/test/chefReadUrPrep
185: Node: viz003
185: 
185: while attempting to start process rank 0.
185: --------------------------------------------------------------------------
185: 4 total processes failed to start
2/3 Test #185: chefReadUrPrep ...................***Failed    0.08 sec
test 186
    Start 186: chefReadRibUrPrep

186: Test command: /usr/local/spack/v0.16/opt/spack/linux-debian8-nehalem/gcc-11.2.0/openmpi-4.1.1-c6dl75o4qks7thuaqfe4d4acbpugbiit/bin/mpirun "-np" "4" "/projects/tools/SCOREC-core/buildCGNS_OneBase/test/chefReadUrPrep" "../../../model.dmg" "bz2:../good_mesh/" "adapt.prerib.inp"
186: Test timeout computed to be: 10000000
186: --------------------------------------------------------------------------
186: mpirun was unable to launch the specified application as it could not access
186: or execute an executable:
186: 
186: Executable: /projects/tools/SCOREC-core/buildCGNS_OneBase/test/chefReadUrPrep
186: Node: viz003
186: 
186: while attempting to start process rank 0.
186: --------------------------------------------------------------------------
186: 4 total processes failed to start
3/3 Test #186: chefReadRibUrPrep ................***Failed    0.08 sec

0% tests passed, 3 tests failed out of 3

Total Test time (real) =   0.27 sec

The following tests FAILED:
	174 - chefStream (Failed)
	185 - chefReadUrPrep (Failed)
	186 - chefReadRibUrPrep (Failed)
Errors while running CTest
Output from these tests are in: /projects/tools/SCOREC-core/buildCGNS_OneBase/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

OK. No worries. Thanks for confirming.

I built everything and tested. I know why CGNS tests fail (he built it for 32bit ints and has exits when that is not true).
Also know why matchedNodeElmReader tests fail (I removed the extra argument...I will fix the test and rerun to be sure this is the only failure). I have no idea why swapDoubles is failing.

187/187 Test #187: chef10 ..............................   Passed    3.59 sec

93% tests passed, 13 tests failed out of 187

Label Time Summary:
SMOKE_TEST    =   1.94 sec*proc (2 tests)

Total Test time (real) = 860.63 sec

The following tests FAILED:
	 17 - swapDoubles (Failed)
	 72 - matchedNodeElementReader_p1 (Failed)
	 73 - matchedNodeElementReader_p4 (Failed)
	123 - cgns_2d_1 (Failed)
	124 - cgns_2d_2 (Failed)
	125 - cgns_2d_3 (Failed)
	126 - cgns_2d_4 (Failed)
	127 - cgns_3d_1 (Failed)
	128 - cgns_3d_2 (Failed)
	129 - cgns_bcs_1 (Failed)
	130 - cgns_bcs_hex (Failed)
	131 - cgns_bcs_2 (Failed)
	132 - cgns_bcs_3 (Failed)
Errors while running CTest
Output from these tests are in: /projects/tools/SCOREC-core/buildCGNS_OneBase/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
Makefile:135: recipe for target 'test' failed
make: *** [test] Error 8

Fixed now

(base) kjansen@viz003: /projects/tools/SCOREC-core/buildCGNS_OneBase $ ctest -R matchedNode
Test project /projects/tools/SCOREC-core/buildCGNS_OneBase
    Start 72: matchedNodeElementReader_p1
1/2 Test #72: matchedNodeElementReader_p1 ......   Passed   41.58 sec
    Start 73: matchedNodeElementReader_p4
2/2 Test #73: matchedNodeElementReader_p4 ......   Passed    8.78 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =  50.38 sec

Here is what swapdoubles does

(base) kjansen@viz003: /projects/tools/SCOREC-core/buildCGNS_OneBase $ ctest -R swap
Test project /projects/tools/SCOREC-core/buildCGNS_OneBase
    Start 17: swapDoubles
1/1 Test #17: swapDoubles ......................***Failed    2.14 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   2.16 sec

The following tests FAILED:
	 17 - swapDoubles (Failed)
Errors while running CTest
Output from these tests are in: /projects/tools/SCOREC-core/buildCGNS_OneBase/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
(base) kjansen@viz003: /projects/tools/SCOREC-core/buildCGNS_OneBase $ ctest -V --rerun-failed 2>&1 |tee -a runfail.log3
UpdateCTestConfiguration  from :/projects/tools/SCOREC-core/buildCGNS_OneBase/DartConfiguration.tcl
UpdateCTestConfiguration  from :/projects/tools/SCOREC-core/buildCGNS_OneBase/DartConfiguration.tcl
Test project /projects/tools/SCOREC-core/buildCGNS_OneBase
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 17
    Start 17: swapDoubles

17: Test command: /usr/local/spack/v0.16/opt/spack/linux-debian8-nehalem/gcc-11.2.0/openmpi-4.1.1-c6dl75o4qks7thuaqfe4d4acbpugbiit/bin/mpirun "-np" "1" "./swapDoubles"
17: Test timeout computed to be: 10000000
17: --------------------------------------------------------------------------
17: mpirun has exited due to process rank 0 with PID 0 on
17: node viz003 exiting improperly. There are three reasons this could occur:
17: 
17: 1. this process did not call "init" before exiting, but others in
17: the job did. This can cause a job to hang indefinitely while it waits
17: for all processes to call "init". By rule, if one process calls "init",
17: then ALL processes must call "init" prior to termination.
17: 
17: 2. this process called "init", but exited without calling "finalize".
17: By rule, all processes that call "init" MUST call "finalize" prior to
17: exiting or it will be considered an "abnormal termination"
17: 
17: 3. this process called "MPI_Abort" or "orte_abort" and the mca parameter
17: orte_create_session_dirs is set to false. In this case, the run-time cannot
17: detect that the abort call was an abnormal termination. Hence, the only
17: error message you will receive is this one.
17: 
17: This may have caused other processes in the application to be
17: terminated by signals sent by mpirun (as reported here).
17: 
17: You can avoid this message by specifying -quiet on the mpirun command line.
17: --------------------------------------------------------------------------
1/1 Test #17: swapDoubles ......................***Failed    2.13 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   2.16 sec

The following tests FAILED:
	 17 - swapDoubles (Failed)
Errors while running CTest
Output from these tests are in: /projects/tools/SCOREC-core/buildCGNS_OneBase/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

(base) kjansen@viz003: /projects/tools/SCOREC-core/buildCGNS_OneBase $ mpirun -np 1 test/swapDoubles (base) kjansen@viz003: /projects/tools/SCOREC-core/buildCGNS_OneBase $ ctest -R swap
Test project /projects/tools/SCOREC-core/buildCGNS_OneBase
Start 17: swapDoubles
1/1 Test #17: swapDoubles ...................... Passed 0.18 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 0.31 sec

after adding an MPI_Finalize().

I think everything (except prior CGNS tests that require 32 bit ints) is now passing so I don't know what that conflict on test/swapDoubles.cc is all about.

I don't know how you want to resolve the prior CGNS tests. Personally, I think they should be removed until someone has the time to make them use cgsize_t to define all integers that interface with CGNS functions as the standard requires. The new file added in this branch, core/phasta/phCGNSgbc.cc has tons of examples of how to do this but it might still be tedious.

I built up a 32 big CGNS library to check to be sure that our code would pick up cgsize_t from the library and use it properly. At this point, the only reader tests we have are cgnsview and paraview. Parview shows the same mesh and cgnsview looks ok. cgnsdiff recognizes the different data types viz.

(base) kjansen@viz003: /projects/tools/SCOREC-core/core/pumi-meshes/phasta/cube_CGNS/sms2mds8Hex/Chef/1-1-Chef (CGNS_tests)$ mpirun -np 1 /projects/tools/SCOREC-core/buildCGNS_32bit/test/chef 
PUMI Git hash 2.2.7
PUMI version 2.2.7 Git hash 364895c4fe9bd0dc522cd423e344ba1389bdb908
"../../geom.smd" and "../../geom.smd" loaded in 0.006355 seconds
mesh ../../mdsMesh/ loaded in 0.001306 seconds
number of tet 0 hex 8 prism 0 pyramid 0
mesh entity counts: v 27 e 54 f 36 r 8
mesh expanded from 1 to 1 parts in 0.000313 seconds
mesh migrated from 1 to 1 in 0.000040 seconds
PARMA_STATUS postSplit disconnected <max avg> 0 0.000
PARMA_STATUS postSplit neighbors <max avg> 0 0.000
PARMA_STATUS postSplit smallest side of max neighbor part 27
PARMA_STATUS postSplit num parts with max neighbors 1
PARMA_STATUS postSplit empty parts 0
PARMA_STATUS postSplit small neighbor counts 1:0 2:0 3:0 4:0 5:0 6:0 7:0 8:0 9:0 10:0 
PARMA_STATUS postSplit weighted vtx <tot max min avg> 27.0 27.0 27.0 27.000
PARMA_STATUS postSplit weighted edge <tot max min avg> 54.0 54.0 54.0 54.000
PARMA_STATUS postSplit weighted face <tot max min avg> 36.0 36.0 36.0 36.000
PARMA_STATUS postSplit weighted rgn <tot max min avg> 8.0 8.0 8.0 8.000
PARMA_STATUS postSplit owned bdry vtx <tot max min avg> 0 0 0 0.000
PARMA_STATUS postSplit shared bdry vtx <tot max min avg> 0 0 0 0.000
PARMA_STATUS postSplit model bdry vtx <tot max min avg> 26 26 26 26.000
PARMA_STATUS postSplit sharedSidesToElements <max min avg> 0.000 0.000 0.000
PARMA_STATUS postSplit entity imbalance <v e f r>: 1.00 1.00 1.00 1.00
max vertex load imbalance of partitioned mesh = 1.000000
ratio of sum of all vertices to sum of owned vertices = 1.000000
max region (3D) or face (2D) load imbalance of partitioned mesh = 1.000000
getGrowthCurves: warning! not implemented for MDS mesh
generated output structs in 0.001839 seconds
mesh mdsMesh/ written in 0.036041 seconds
geombc file written in 0.001091 seconds
matchface 4, 4CGNS file written in 0.053927 seconds
  - verifying tags: mapping_vtxid_ver, mapping_partid_ver
mesh verified in 0.000634 seconds
(base) kjansen@viz003: /projects/tools/SCOREC-core/core/pumi-meshes/phasta/cube_CGNS/sms2mds8Hex/Chef/1-1-Chef (CGNS_tests)$ ls
1-procs_case  adapt.inp  chefOut.cgns  correct.cgns  mdsMesh  val11.Log  val2.Log  val3.Log  val4.Log  val5.Log  val6.Log  val7.Log  val8.Log  val.Log  vg.21751  vg.2872  vg4398  vg4401  vg.5025  vg.5026  vg.6064  vg.6065  vg.7028  vg.7029  vg.7932  vg.7933
(base) kjansen@viz003: /projects/tools/SCOREC-core/core/pumi-meshes/phasta/cube_CGNS/sms2mds8Hex/Chef/1-1-Chef (CGNS_tests)$ cgnsdiff correct.cgns chefOut.cgns 
/Base/Zone <> /Base/Zone : data types differ
/Base/Zone/Hex/ElementConnectivity <> /Base/Zone/Hex/ElementConnectivity : data types differ
/Base/Zone/Hex/ElementRange <> /Base/Zone/Hex/ElementRange : data types differ
/Base/Zone/Quad/ElementConnectivity <> /Base/Zone/Quad/ElementConnectivity : data types differ
/Base/Zone/Quad/ElementRange <> /Base/Zone/Quad/ElementRange : data types differ
/Base/Zone/ZoneBC/SurfID_1/PointList <> /Base/Zone/ZoneBC/SurfID_1/PointList : data types differ
/Base/Zone/ZoneBC/SurfID_2/PointList <> /Base/Zone/ZoneBC/SurfID_2/PointList : data types differ
/Base/Zone/ZoneBC/SurfID_3/PointList <> /Base/Zone/ZoneBC/SurfID_3/PointList : data types differ
/Base/Zone/ZoneBC/SurfID_4/PointList <> /Base/Zone/ZoneBC/SurfID_4/PointList : data types differ
/Base/Zone/ZoneBC/SurfID_5/PointList <> /Base/Zone/ZoneBC/SurfID_5/PointList : data types differ
/Base/Zone/ZoneBC/SurfID_6/PointList <> /Base/Zone/ZoneBC/SurfID_6/PointList : data types differ
/Base/Zone/ZoneGridConnectivity/Periodic Connectivity/PointList <> /Base/Zone/ZoneGridConnectivity/Periodic Connectivity/PointList : data types differ
/Base/Zone/ZoneGridConnectivity/Periodic Connectivity/PointListDonor <> /Base/Zone/ZoneGridConnectivity/Periodic Connectivity/PointListDonor : data types differ
(base) kjansen@viz003: /projects/tools/SCOREC-core/core/pumi-meshes/phasta/cube_CGNS/sms2mds8Hex/Chef/1-1-Chef (CGNS_tests)$ cd ../2-1-Chef/
(base) kjansen@viz003: /projects/tools/SCOREC-core/core/pumi-meshes/phasta/cube_CGNS/sms2mds8Hex/Chef/2-1-Chef (CGNS_tests)$ mpirun -np 2 /projects/tools/SCOREC-core/buildCGNS_32bit/test/chef 
PUMI Git hash 2.2.7
PUMI version 2.2.7 Git hash 364895c4fe9bd0dc522cd423e344ba1389bdb908
"../../geom.smd" and "../../geom.smd" loaded in 0.005676 seconds
mesh ../../mdsMesh/ loaded in 0.000963 seconds
number of tet 0 hex 8 prism 0 pyramid 0
mesh entity counts: v 27 e 54 f 36 r 8
planned Zoltan split factor 2 to target imbalance 1.010000 in 0.000733 seconds
mesh expanded from 1 to 2 parts in 0.000577 seconds
mesh migrated from 1 to 2 in 0.002842 seconds
PARMA_STATUS postSplit disconnected <max avg> 0 0.000
PARMA_STATUS postSplit neighbors <max avg> 1 1.000
PARMA_STATUS postSplit smallest side of max neighbor part 9
PARMA_STATUS postSplit num parts with max neighbors 2
PARMA_STATUS postSplit empty parts 0
PARMA_STATUS postSplit small neighbor counts 1:0 2:0 3:0 4:0 5:0 6:0 7:0 8:0 9:2 10:0 
PARMA_STATUS postSplit weighted vtx <tot max min avg> 36.0 18.0 18.0 18.000
PARMA_STATUS postSplit weighted edge <tot max min avg> 66.0 33.0 33.0 33.000
PARMA_STATUS postSplit weighted face <tot max min avg> 40.0 20.0 20.0 20.000
PARMA_STATUS postSplit weighted rgn <tot max min avg> 8.0 4.0 4.0 4.000
PARMA_STATUS postSplit owned bdry vtx <tot max min avg> 9 9 0 4.500
PARMA_STATUS postSplit shared bdry vtx <tot max min avg> 18 9 9 9.000
PARMA_STATUS postSplit model bdry vtx <tot max min avg> 34 17 17 17.000
PARMA_STATUS postSplit sharedSidesToElements <max min avg> 1.000 1.000 1.000
PARMA_STATUS postSplit entity imbalance <v e f r>: 1.00 1.00 1.00 1.00
malloc used before Bfs: min 4.796875 max 4.957031 avg 4.876953 imb 1.016420
malloc used before reorder: min 4.796875 max 4.957031 avg 4.876953 imb 1.016420
mesh reordered in 0.000378 seconds
malloc used after reorder: min 4.796875 max 4.957031 avg 4.876953 imb 1.016420
max vertex load imbalance of partitioned mesh = 1.000000
ratio of sum of all vertices to sum of owned vertices = 1.333333
max region (3D) or face (2D) load imbalance of partitioned mesh = 1.000000
getGrowthCurves: warning! not implemented for MDS mesh
generated output structs in 0.001380 seconds
mesh mdsMesh/ written in 0.001702 seconds
geombc file written in 0.000846 seconds
matchface 4, 4CGNS file written in 0.072810 seconds
  - verifying tags: mapping_partid_ver, mapping_vtxid_ver
mesh verified in 0.001736 seconds
(base) kjansen@viz003: /projects/tools/SCOREC-core/core/pumi-meshes/phasta/cube_CGNS/sms2mds8Hex/Chef/2-1-Chef (CGNS_tests)$ cgnsdiff correct.cgns chefOut.cgns 
/Base/Zone <> /Base/Zone : data types differ
/Base/Zone/Hex/ElementConnectivity <> /Base/Zone/Hex/ElementConnectivity : data types differ
/Base/Zone/Hex/ElementRange <> /Base/Zone/Hex/ElementRange : data types differ
/Base/Zone/Quad/ElementConnectivity <> /Base/Zone/Quad/ElementConnectivity : data types differ
/Base/Zone/Quad/ElementRange <> /Base/Zone/Quad/ElementRange : data types differ
/Base/Zone/ZoneBC/SurfID_1/PointList <> /Base/Zone/ZoneBC/SurfID_1/PointList : data types differ
/Base/Zone/ZoneBC/SurfID_2/PointList <> /Base/Zone/ZoneBC/SurfID_2/PointList : data types differ
/Base/Zone/ZoneBC/SurfID_3/PointList <> /Base/Zone/ZoneBC/SurfID_3/PointList : data types differ
/Base/Zone/ZoneBC/SurfID_4/PointList <> /Base/Zone/ZoneBC/SurfID_4/PointList : data types differ
/Base/Zone/ZoneBC/SurfID_5/PointList <> /Base/Zone/ZoneBC/SurfID_5/PointList : data types differ
/Base/Zone/ZoneBC/SurfID_6/PointList <> /Base/Zone/ZoneBC/SurfID_6/PointList : data types differ
/Base/Zone/ZoneGridConnectivity/Periodic Connectivity/PointList <> /Base/Zone/ZoneGridConnectivity/Periodic Connectivity/PointList : data types differ
/Base/Zone/ZoneGridConnectivity/Periodic Connectivity/PointListDonor <> /Base/Zone/ZoneGridConnectivity/Periodic Connectivity/PointListDonor : data types differ

as I am not all that familiar with cgnsdiff, I also applied it to two hex meshes make with the 64 bit version. In that case it caught that the arrays sizes were different and the data types were the same. This is not a proof that the 32 bit is correct but it at least produced arrays of the same size and number. This plus paraview is pretty convincing.

HMMM. All the tests pass. I was expecting to get fails but I guess we we are not checking that its output is empty????

Would it make sense to change the test to something like this
cgnsdiff correct.cgns chefOut.cgns |grep -v 'data types differ'

and then add some check that it is actually an empty result? Googling cgnsdiff I find it is only checking headers: label, type and dimensions

HMMM. All the tests pass. I was expecting to get fails but I guess we we are not checking that its output is empty????

Would it make sense to change the test to something like this cgnsdiff correct.cgns chefOut.cgns |grep -v 'data types differ'

and then add some check that it is actually an empty result?

Yeah, it's probably only checking that the command exits with a non-zero exit code. So doing something like if [[ $(cgnsdiff correct.cgns chefOut.cgns) ]] then; return 1; fi should work.

Googling cgnsdiff I find it is only checking headers: label, type and dimensions

Interesting. Perhaps h5diff would be better to do then if that's the case.

vs. the same command on two files with the same data types but different actual data

(base) kjansen@viz003: /projects/tools/SCOREC-core/core/pumi-meshes/phasta/cube_CGNS/sms2mds8Hex/Chef/1-1-Chef (CGNS_tests)$ h5diff correct.cgns  ../../../sms2mdsAllHex/Chef/1-1-Chef/correct.cgns 
dataset: </Base/Zone/ data> and </Base/Zone/ data>
2 differences found
dataset: </Base/Zone/Hex/ElementRange/ data> and </Base/Zone/Hex/ElementRange/ data>
1 differences found
dataset: </Base/Zone/Quad/ElementRange/ data> and </Base/Zone/Quad/ElementRange/ data>
2 differences found
dataset: </Base/Zone/User Data/nCoordsOnRank/ data> and </Base/Zone/User Data/nCoordsOnRank/ data>
1 differences found
dataset: </Base/Zone/User Data/nHexOnRank/ data> and </Base/Zone/User Data/nHexOnRank/ data>
1 differences found
dataset: </Base/Zone/User Data/nQuadOnRank/ data> and </Base/Zone/User Data/nQuadOnRank/ data>
1 differences found
dataset: </Base/Zone/ZoneGridConnectivity/Periodic Connectivity/GridConnectivityProperty/Periodic/Translation/ data> and </Base/Zone/ZoneGridConnectivity/Periodic Connectivity/GridConnectivityProperty/Periodic/Translation/ data>
2 differences found
--------------------------------
Some objects are not comparable
--------------------------------
Use -c for a list of objects.

To be clear, cgnsdiff has flags to request a comparison of data but perhaps h5diff is a better choice anyway since it is more likely to be available?