Reference no: EM133440865

File Sharing Cloud

1 Task overview
Realize a functional distributed system that will ensure work with ASCII coded text files. The system allows the user the following:
- Adding a new file with a unique name and path to the system.
- Fetching an arbitrary file from a distributed system.
- Deleting the file, on the local server.
- The topological organization of the system enables faster retrieval of files.
- Resistance to failures.

The project can be implemented in any programming language, as long as it satisfies all functional and non-functional requirements. It is allowed to use a distributed system with exercises as a starting point. In order to be awarded any points, it is necessary that the implemented functionalities work on a distributed system.

The functional requirements for the system are described in section 2. Non-functional system requirements are described in section 3.

2 Functional requirements

Basic functionalities
The File Sharing Cloud system serves to provide work with ASCII coded text files. Files should be organized in a virtual file structure, where an arbitrary subset of this entire structure can be present on each individual node. Interaction with the system is done through the command line (CLI). The virtual file system should be preserved as long as there is at least one active node. When the last node shuts down, the virtual file system also ceases to exist, and it should not be possible to reconstruct it afterwards.

Basic functionalities for the system include:
- Add - Adding a new file to the system.
- Pull - Fetching a file that is not currently present on the node.
- Remove - Removing a file from the system.
There should be a directory on the local system that is initially empty, and which is used when working with the system:
- Working root - where the files that the user wants to actively work with will be located.

Node configuration
When starting a node, a configuration file is automatically read that lists the following attributes:
- Working root - the path on the local system where the work files are stored. (string)
- The port on which the node will listen. (short)
- IP address and port of the bootstrap node - section 3.1. (string and short)
- Weak failure limit - section 3.2. (int)
- Strong failure limit - section 3.2. (int)
It is assumed that all nodes will have matching configuration files, and there is no need to check further if this is the case. It is allowed that the system does not work due to an incorrectly set configuration file.

Commands

The user can issue the following commands to the system:
- add name - Adds the file name to the system. A directory can also be specified as name, in which case its entire contents are recursively added to the system. If the virtual system already contains a file with this name in this path, report an error and do not add the file to the system.

- pull name - Retrieves the file name from the system. The file name is specified as the full path to the file and its name.

- remove name - Removes the file name from the system. A directory can also be specified as name, in which case the remove operation is performed recursively for the entire directory.

- stop-Shutdown of the node.

With all commands, when specifying the file name, a path relative to the working root, which is specified in the configuration file, is expected. Filenames will never have spaces, and there is no need to support them.

3 Non-functional requirements

System architecture
It is allowed to have a bootstrap server, which is not a node in the network (ie, all notes in this document that refer to nodes do not refer to a bootstrap server). The following assumptions apply to bootstrap:
- It is used exclusively for the first inclusion of a node in the network. As soon as bootstrap sends the new node the address and port of a node from the system, communication with bootstrap is terminated.
- Bootstrap server has very limited throughput. Communication with the bootstrap server must be minimized.
- It is not allowed for the bootstrap server to be aware of the system architecture, and to be the one to organize it. The system must be self-organizing.

It should be possible to power on and off nodes at any time during system operation, including while other nodes are being powered on or off.

There are two variants for system architecture that are scored differently:
- Simple graph (100% points) - since the graph is not complete, in order for node A to forward a message to node B, it must find a (not necessarily complete) path through the system to node B. Here, it is necessary that the number of hops between A and B tends to logarithmic dependence on the total number of nodes. If the number of jumps between arbitrary A and B tends to a linear dependence, the implementation is scored as if a complete graph was made (50% points). In order for a graph to be considered simple, the number of neighbors for all nodes must have a logarithmic dependence on the total number of nodes. There must not be a central point of failure (node after which the system stops working). There must not be a bottleneck - for the purposes of this project, we define a bottleneck as a node (or multiple nodes with a fixed number) through which communication often flows. If there are nodes through which communication often flows, but their number depends on the number of nodes in the system, so that communication is naturally distributed among them, then they are not bottlenecks. Starting new nodes, as well as stopping active nodes can and should cause system restructuring.

- Complete graph (50% points) - every node is connected to every other node. Communication is always direct.

Failure detection
Node failure is detected in two phases. As a constant system parameter, there should be a weak failure limit (eg 1000ms) and a strong failure limit (eg 10000ms), both given as the time the node does not respond to ping.

If a node exceeds the weak limit of failure, we mark it as suspicious, but we do not start removing the node and restructuring the system, but ask another stable node to confirm that the suspicious node is really problematic.

If we receive confirmation that the node is suspicious, and then the strong failure limit expires, the node is eliminated from the system and restructuring begins.

When a node fails, its previous work must not be lost. If there is a free node, it should take over the data that the failed node was responsible for. If there is no free node, it is necessary to restructure the system so that every part of the file structure is present on some node.

The system should be able to cope with a "worst case" situation in which an intelligent malicious attacker can search and simultaneously take down any two nodes every five minutes.

Arrangement of data in the system
The files are located on the node where they were created. And there must be a copy of that file somewhere. Basically, some back up.

General non-functional requirements
The system must operate on a real network, where each message has an arbitrary delay and the channels are non-FIFO. If the system is tested on a single machine, it is necessary to introduce artificial delays when sending messages, for realistic testing. Even if testing is done on a single machine, it is not allowed to fix the destination IP address when sending the message as "localhost".

All communication must be explicitly defined and documented. If a node receives a message that does not follow the protocol, it should be discarded and ignored. The documentation of the protocol should contain at least everything that is necessary to write a new server that will participate in the operation of the system. Typically, it is a list of all the messages that exist in the system and their format - the order of the values that are passed, their type and their meaning. If there is a specific order of sending messages, then specify that as well.