Apache Arrow provides a Dataset
API, which acts as an abstraction over a collection of files in different storage backends like S3 and HDFS. It supports different file formats like CSV and Parquet through the FileFormat
API. In SkyhookDM, we create a new file format called SkyhookFileFormat
on top of ParquetFileFormat
, which besides providing all the features of Parquet allows offloading file fragment scan operations into the storage backend. Offloading scan operations increases the query performance many folds, provides better scalability, and results in less network traffic.
NOTE: Please make sure docker and docker-compose is installed.
- Clone the repository. The default branch gets cloned.
git clone https://github.com/uccross/arrow
- Run the
ubuntu-cls-demo
step in the docker-compose file. This step will start a single node Ceph cluster inside the container, mount CephFS, put sample data into CephFS, and open an example Jupyter notebook with PyArrow installed.
cd arrow/
docker-compose run --service-ports ubuntu-cls-demo
-
For installing SkyhookDM-Arrow with Rook on Kubernetes, check out this guide.
-
For installing SkyhookDM-Arrow on CloudLab, check out this guide. To deploy SkyhookDM on bare-metal in general, check out this guide.
-
Enables pushing down filters, projections, compute operations to the Storage backend for minimal data transfer over the network.
-
Allows storing data in Parquet files for minimizing Disk I/O though predicate and projection pushdown.
-
Plugs-in seamlessly into the Arrow Dataset API and leverages all its functionality like dataset discovering, partition pruning, etc.
-
Minimal overhead in requirements:
- Requires CephFS to be mounted.
- Requires using the
SplittedParquetWriter
API to write arrow Tables.
-
Built on top of latest Ceph v15.2.x.
-
file_skyhook.h
: This file contains the definitions of 3 APIs. TheRadosConnection
,SkyhookDirectObjectAccess
, and theSkyhookFileFormat
. TheRadosConnection
API helps create a connection to the Ceph cluster and provides a handle to the cluster that can be passed around. TheSkyhookDirectObjectAccess
API provides abstractions for converting filenames in CephFS to object IDs in the Object store and allows interacting with the objects directly. TheSkyhookFileFormat
API takes in the direct object access construct as input and contains the logic of pushing down scans to the underlying objects that make up a file. This file also contains functions for (de)serializing scan options and query results intoceph::bufferlist
using Flatbuffers for sending them over the network. -
rados.h
: Contains a wrapper for thelibrados
SDK for exposinglibrados
methods likeinit2
,connect
,stat
,ioctx_create
, andexec
which are required for establishing the connection to the Ceph cluster and for operating on objects directly.
-
_rados.pyx
: Contains Cython bindings to theSkyhookFileFormat
C API. -
rados.py
: This file contains the definition of theSplittedParquetWriter
. It is completely implemented in Python.
cls_arrow.cc
: Contains the Rados objclass functions and APIs for interacting with objects in the OSDs. Also, it includes aRandomAccessObject
API to give a random access file view of objects for allowing operations like reading byte ranges, seeks, tell, etc.
Check out these instructions for setting up a local development environment.