Day 38 2SSL File Transfer Fleshout

What I did: Continued working on file transfer fleshout

Time spent: 88.967

Comments:

There are two components we need to understand:

1) The transport and session layer.
2) The actual data transfer layer.

Allows transferring an arbitrary data up to 34PiB in 1MiB chunks (ish).

The session layer is separated from the transport layer. This is so that the session can be managed separately from 2SSL connection which doesn't make sense until you look at the data center structure. What occurs here is that session information is attached internally and then sent to stateless 'workers'. This means that workers have everything that they need directly on the request. Meaning that if one data center's servers are overloaded they can still process ingress and have the data session attached to the request, then sent over an internal network to the other data center. While at the same time migrating the 2SSL termination data and keys over to less inundated data centers. This allows a phased transfer at all three layers:

1) 2SSL layer
2) Session layer
3) Actual file or raw data layer.

The third is the most interesting. What happens is that if a new UUID that hasn't been seen before is used, it'll accept the new file chunk and then send a request for the meta data of that file (which is stored in the first chunk). To another data center. Since this information is derived from the file size and the SPacket size we know how many packets are going to be used. This allows different file chunks to be uploaded via different paths to different data centers, being aggregated eventually at the end, with minimal cross data center communication. This information can exist easily within an SPacket command (160 bytes). This ensures secure inter data center communication in the event that internal communication lines or private communication lines can't be trusted.

Furthermore these do not have to be transfered in any particular order (aside from the requirement that the packet containing the file meta needs to be transferred first).