This thread is to discuss how fortran ordering should be implemented in legate. After meeting with @supunkamburugamuve and doing some more research I have come to the conclusion that HDF5 does not have any mechanisms for natively supporting or indicating to the user that an array was stored as column-major. HDF5 fundamentally assumes row-major ordering.
Single Language Case
The standard solution based on my research is to swap the dimensions when calling the HDF5 C-API (read & write). There is no way beyond a custom flag to indicate the true dim ordering of the file.
Here is some relevant code from the HDF5 Fortran API which swaps the dimensions before calling the C-API.
From the Report Data Set Dimensions section of the MATLAB HDF5 docs
The MATLAB low-level HDF5 functions report data set dimensions and the shape of data sets differently than the MATLAB high-level functions. For ease of use, the MATLAB high-level functions report data set dimensions consistent with MATLAB column-major indexing. To be consistent with the HDF5 library, and to support the possibility of nested data sets and complicated data types, the MATLAB low-level functions report array dimensions using the C row-major orientation.
Language Interoperability
I think there will just always be a fundamental issue when crossing language boundaries. You can fix it with a transpose (overhead), but the main issue is that there is no way to know the true dimension ordering of the file you read. Here is a small example with Python and MATLAB.
import h5py
import numpy as np
A = np.array([[11, 12, 13],
[21, 22, 23]])
with h5py.File("order.h5", "w") as f:
f["A"] = A
print(A)
# [[11 12 13]
# [21 22 23]]
print(A.shape)
# (2, 3)A = h5read("order.h5", "/A");
disp(A)
% 11 21
% 12 22
% 13 23
disp(size(A))
% 3 2
This thread is to discuss how fortran ordering should be implemented in legate. After meeting with @supunkamburugamuve and doing some more research I have come to the conclusion that HDF5 does not have any mechanisms for natively supporting or indicating to the user that an array was stored as column-major. HDF5 fundamentally assumes row-major ordering.
Single Language Case
The standard solution based on my research is to swap the dimensions when calling the HDF5 C-API (read & write). There is no way beyond a custom flag to indicate the true dim ordering of the file.
Here is some relevant code from the HDF5 Fortran API which swaps the dimensions before calling the C-API.
From the
Report Data Set Dimensionssection of the MATLAB HDF5 docsLanguage Interoperability
I think there will just always be a fundamental issue when crossing language boundaries. You can fix it with a transpose (overhead), but the main issue is that there is no way to know the true dimension ordering of the file you read. Here is a small example with Python and MATLAB.