The network topology, data flow and protocols
The network topology for the broadcast service
The broadcast service provides the availability of live data to several destinations. Usually, each destination should be the specific LAN broadcast address. The description of the video content and the port number to view the broadcast together are named as channel. Each client located on this particular local network may display the video and closed captioned text on the screen. But each client is free to switch to another channel or playback the old video selected from the database. The client may play only single video at the time.
The problem: consuming network resources. Transmitting the live video data over the network means sending approximately 100 UDP packets per second to this LAN broadcast address with typical data rate 150 K/s. That consume resources of network and routers. This immediately imply that the Video Server must provide the mechanism to transmit data only to those destinations where it finds at least one interested client. Second, the video data itself should be replicated over different network storages to minimize unnecessary traffic.
Dynamic vs. static topology. The dynamic topology means that the explicit command from the user interested to view the channel selected from the list activates the selection of push server which may or already is the transmitter for this channel. The activation of the previously inactive transmitting push server means that the topology have been changed. In fact, acquisition server must add another push server to the list of data recipients, database server which manage all the topology must add this LAN as active recipient of the channel.
Design decision. As opposite to the dynamic topology the static one was selected as the design decision. Let describe it and let show that this simplification does not decrease the set-up and use flexibility.
The static topology. Database server loads the network topology from properties as the static graph. The initial source of broadcast video stream is the acquisition server. That data availability should appear to the final destination, client, as the description and the port number to listen for UDP data packets: we define this as the channel. Thus, the particular acquisition server is the only one source of the particular channel. At the same time, the particular channel is coming only from this acquisition server and have no way to be sourced from anything else. This does not restrict that different acquisition servers may receive the same TV program - at that case two absolutely different channels may have the same description and the same content while been defined as different channels and have no way to appear to the client on the same UDP port. Thus, the unique channel ID is assigned to each channel and may appear in any program logic to specify it. The acquisition server does not knows the final destinations of the broadcasting and never loads this information. Instead, at the boot time it receives from the database server the list of push servers that will (may) work as transmitters and will (may) store the data locally.
The push server should be located on the computer that have a convenient access to all local networks where it should broadcast. In case if the broadcast destination is the multicast address this computer should support multicasting. The push server may have only one connection with acquisition server to receive data, but it may have several broadcast destinations. This does not restrict the flexibility to have multiple channels to be broadcasted by this computer because several push servers may coexist at the the same computer. The database server loads from properties the list of all push servers with the channel ID assigned to the each one. For each push server it loads also the list of all broadcast destinations. So given the client network address we may get the unique list of all channels that are (or may be) broadcasted to this user. In current implementation the simple algorithm is used and it will not work properly with multicast addresses.
The static topology should not imply that all data are broadcasted permanently in all possible destinations. This problem is resolved requiring clients to acknowledge the interest of the channel to transmitter, i.e. the push server. If given some fixed timeout the push server had not received any acknowledge of interest from the specific destination, this channel should be suspended. We should not confuse the suspended transmission of channel with inactive channel. While the channel is suspended the network topology is not affected. The database server that controls the graph of the network is not interested about the details of transmit-ion. From the other side the client distinguish only active and inactive channels, but don't know if the transmission is suspended. When the client wants to receive the channel, it starts the periodical posting of packets of interest to the push server. Its the push server business to change the status of transmission to not suspended.
What is active and inactive channel? The static network topology does not imply that all components are required to operate, been located in permanently accessible networks on computers that never crash or reboot. Instead, we define the status of the channel relatively to the particular destination. The status may be active or inactive. If the acquisition server does not operates or is unreachable, its channel ID may not be used by another source, so this channel is defined to be inactive relatively to all destinations. If the particular local network is not listed as the destination in any push server destinations list, this channel is not listed by request of any client from that network and its status is undefined. If the acquisition server successfully transmit data to some push servers while some others are not operating, the status of this channel will be different relatively different clients. If several push servers may transmit the same channel to the same destination, this is not the error, but that means that user may not distinguish which push server it wants to listen because by definition this channel have only one ID. So one of push servers will be selected. If one push server is operating while another does not work by some reason, the program must ensure that only the operating one is selected.
The network topology for the playback service
That case is rather simple. Playback stream is the point-to-point UDP stream: from push server to the client. When client requests the movie to play, the database server returns the parameters of the nearest push server to serve the request. Then client sends the "play movie" command directly to the push server.
Start-up order of components
All components in Video server talk to each other by various protocols using TCP and UDP sockets. The destination for any particular service may be unavailable at the present time. But all connections in the system are subject to reconnect after the timeout. At the same time, if the service accept connections and this particular connection is lost by any reason, the service is returned back into waiting state.
So the start-up order of components in fact is not important to get the system functional. But this topic describes the order in with connections are established - the way to eliminate all reconnects.
First, start-up all push server. When started, push server quietly waits for incoming commands.
The real squid of program is the database server , which holds all the metadata, system setup and manage the data flow of all the system. database server should start-up the second. Then, acquisition server starts and try to setup the connection to database server. On success, it receives the address and the port number of push server, which will be responsible to broadcast data among specified networks. If the connection fails, acquisition server will repeat the connection after the timeout.
When database server starts, it should post the command to push server to configure the list of networks and the port number to broadcast.
If no acquisition servers are registered, the database server will assume that we work in off-line mode and broadcast is not available. When the first acquisition server registers itself, the live broadcasting is available to any client.
When on-line mode is (re)established, Database server posts to Acquisition server (with protocol 5 - look next section) the address of push server to which it must post all data (protocol 6). If acquisition server cannot connect to it, it must post the error notification back and drop the connection.
Data stream
Fig 1: all protocols and default port numbers. Click here for animated picture
1. Acquisition server - database server TCP protocol. Initiated by the acquisition server. It registers itself, request address for immediate broadcasting and periodically checks for new commands.
2. Client - database server TCP protocol. Initiated by the client. First, lets the client to acquire all sort of data from Database server. Second, if client passed the authorization and is operated in superuser mode, lets it to configure all servers and lets the client to post some commands to a acquisition server.
3. Acquisition server to push( broadcast ) server TCP protocol. It uses the same data format as broadcast protocol #x, except it is TCP and there are only one destination.
4. Push server UDP control protocol. All other components use this UDP connection to post commands to push server.
5. Individually requested UDP data stream.
6. Permanently broadcasted UDP data stream.