A Distributed Pipeline, which aims to provide an infrastructure to connect the hardware resources and divide the ’pipelinable tasks’ into stages and process them in parallel and in order, synonymous to an assembly line of a software task.

• About
• Getting Started
• Deployment
• Usage
• Built Using
• Contributing
• Authors

Hyper Line is a generic pipeline architecture for distributed system environment which can run any application that can be divided into arbitrary number of sub-tasks such that the each sub-task running on a different worker node and can send its output to another worker node running another sub-task and utilizing the output from the previous stage as input for itself. The final output is returned to the server which in turn passes it to the user of the application.

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

This pipeline runs on Linux Machines. That is the only prerequisite for this project

To download and build the project, execute the following commands at the src drirectory the project directory

	git clone https://github.com/vpatel95/hyper-line
	cd hyper-line/src
	make clean        
	make

This will generate binaries for all the components in the build directory

Two applications have been included in the project for the purpose of testing.

• Symmetric Key Encryption
• Sequential Arthmetic Operation

There are three components of the system. Each component will run on seperate machines.

To run the server follow the given steps.

1. Create a config file server.json from the template
   • log_level : There are 5 level of logs namely DEBUG=0, INFO=1, WARN=2, ERROR=3, FATAL=4
   • log_quiet : Set this to false if you want to print the logs in the console along with the file
   • addr : Ip address on which the server will be running
   • uport : Port number for user server
   • wport : Port number for worker server
   • A worker config template is given below

			{
				"server": {
					"log_user_file": "<log_file>",
					"log_worker_file": "<log_file>",
					"log_level": 1,
					"log_quiet": true,
					"addr": "<ip_address>",
					"uport": <user_port>,
					"wport": <worker_port>
				}
			}

Run the server using ./avd_server <config_file>

To run the user application on pipeline user needs to follow the given steps.

1. Compile the application in form of shared object.
   • Create a Makefile in the application directory
   • Set the following mandatary variables
     • ROOT : Set the relative path to the src directory of the project
     • TARGETS : Set the name of binary you want to generate. Keep the extension as .so
   • Set following optional variables if required
     • LIB_DIR : set this with directory of source files if residing outside the current directory
     • CUSTOM_LIBS : List down the source files in the directories in LIB_DIR
     • LIBS : libraries to be linked dynamically, e.g -lm or -pthread
     • INCLUDE : Set this to directory where header files are residing, only if outside current directory
   • In the end of file add the line include $(ROOT)/common.mk
   • When you run make in the src directory, application will also be build along with other components and stored in build directory
2. Create a config file user.json from the template
   • uname : username chosen by user
   • log_level : There are 5 level of logs namely DEBUG=0, INFO=1, WARN=2, ERROR=3, FATAL=4
   • log_quiet : Set this to false if you want to print the logs in the console along with the file
   • file : binary file of the application
   • input : input file for the application
   • output : file where output is to be stored
   • num : number of stage in the chain, First worker being num=1, second being num=2 and so on
   • func : This is the name of function for the stage in the shared object binary
   • A user config template is given below

			{
				"user" : {
					"uname": "<username>",  
					"log_file": "<log_file>",  
					"log_level": 1,  
					"log_quiet": true,  
					"srvr_addr": "<server_ip_address>",  
					"srvr_port": <server_user_port>,  
					"num_tasks": 1,  
					"tasks": [{
						"name": "<task name>",
						"file" : "<task binary file>",
						"input" : "<task input file>",
						"output" : "<task output file>",
						"num_stages": 2,
						"stages": [{
							"num": 1,
							"func": "<stage 1 function>"
						}, {
							"num": 2,
							"func": "<stage 2 function>"
						}]
					}]
				}
			}

Run the user using ./avd_user <config_file>

To run the worker follow the given steps.

1. Create a config file worker.json from the template
   • uname : username chosen by user
   • log_level : There are 5 level of logs namely DEBUG=0, INFO=1, WARN=2, ERROR=3, FATAL=4
   • log_quiet : Set this to false if you want to print the logs in the console along with the file
   • addr : Ip address on which the peer server will be running
   • peer_port : Port number for peer server
   • A worker config template is given below

			{
				"worker" : {
					"uname": "<username>",
					"log_file": "<log_file>",
					"log_level": 1,
					"log_quiet": true,
					"addr" : "<peer_server_ip_address>",
					"peer_port": <peer_server_port>,
					"srvr_addr": "<server_ip_address>",
					"srvr_port": <server_worker_port>
				}
			}

Run the worker using ./avd_worker <config_file>

It does not require any special steps for deployment. Same steps for Installing and Usage can be followed.

• [C] - Programming Language

• @vpatel95

See also the list of contributors who participated in this project.