-We have designed P2P-Tube as a video sharing platform based on NextShare video streaming capabilities. Currently there are two browser plugins for video rendering developed by P2P-Next to be NextShare compliant. The first of them, known as SwarmPlayer, is based on HTML5 video tag. Because HTML5 is still a draft and older browsers do not support it, another NextShare plugin based on VLC has been created, named NextSharePC. P2P-Tube supports both versions of NextShare plugins and users can choose which to use directly from the watching page.
+Besides the need to incorporate Next-Share technologies, P2P-Tube was also designed to be useful to the users. In the next subsections the usability and technical needs that governed our architecture are explained.
-Within the platform video assets can be grouped in categories, facilitating easier browsing. Searching by keywords is also possible, both in the whole set of videos or in a specific category. Search results are ordered by relevance. Users can sign up for an account and singing in is possible with internal P2P-Tube authentication, with LDAP or with OpenID. Logged in users can comment video assets and once a day for each video can rate it with a Like or a Dislike.
+\subsection{Design Principles}
+\label{subsec:design-principles}
+
+P2P-Tube platform has been designed with a few basic design principles in mind:
+
+\begin{enumerate}
+ \item Provide as much as possible the \textbf{features} needed by such a platform, as they appear on similar platforms from the web.
+ \item Provide to users a familiar \textbf{user interface} which resembles similar platforms from the web.
+ \item Ensure that the platform \textbf{scales} well to a size of a few thousands of users.
+ \item Ensure that the platform is \textbf{general purpose}.
+ \item Make the platform available for \textbf{free} and distribute the code as \textbf{open-source}, encouraging community contributions.
+\end{enumerate}
+
+Being developed in academia, in University Politehnica of Bucharest, P2P-Tube was designed to fit well to our needs. The scalability requirements of a few thousands users, enumerated in Principle 3 above is enough for our Faculty of Automatic Control and Computers. However, despite this product was designed in the academia and fits its needs, we ensure that it can be generally purpose as stated in Principle 4 because of the features incorporated and its user interface tied to Principles 1 and 2.
+
+\subsection{Features}
+\label{subsec:features}
+
+P2P-Tube provides to the users the following features:
+
+\begin{itemize}
+ \item \textbf{Categories}. Video assets can be grouped in \textit{categories} for a better organization of them.
+ \item \textbf{Video plugin switching}. If the user has installed both Next-Share plugins (NextSharePC and SwarmPlayer) he is able to switch between them at any moment from the video watching page.
+ \item \textbf{Users and accounts}. Users can apply for an account by registering to the site. Throughout registration, several fields can be completed, some of them mandatory, some of them optional, defining the \textit{user profile}. Authentication is possible through three methods: \textit{internal authentication}, \textit{LDAP authentication} and \textit{OpenID authentication}.
+ \item \textbf{Uploading video assets}. Logged in users can upload new video assets to the site, which can be seen by others.
+ \item \textbf{Social interaction}. Users can see each other's profiles, comment each other's video assets and vote them. Video assets uploaded by an user can be explored.
+ \item \textbf{Searching}. Video assets can be searched by keywords and the results must be ordered by their relevance. The search can be done in the whole set of videos or in a particular category.
+\end{itemize}
+
+\textit{Internal authentication} is for users that applied for an account by the \textit{Registration} page. There are situations when the platform must be integrated with other services or sites. For example in our university we use P2P-Tube to share videos of technical presentations, courses or social events. We also have Moodle learning platform where students already have an account, so it was desired that users can authenticate on P2P-Tube we the same account credentials from Moodle. This has been achieved thorough LDAP which permits students to authenticate on both sites with the same user name and password, without requiring them to create an additional account. Another way to make users skip signing up is by using OpenID, a standard protocol which allows them to sign in through an account from another site. However, this third party site must support OpenID. So, for example, users that already have an account on Yahoo!, Google, Twitter or WordPress can securely sign in with the same user names and passwords from that accounts. When users log in for the first time using external authentication, like LDAP or OpenID, they must complete their profile.
Users can upload video assets through an upload mechanism described in the next subsection.
\subsection{Content Ingestion}
\label{subsec:content-ingestion}
-NextSharePC can play any video format as long as VLC supports it. SwarmPlayer supports only video containers and codecs which are compliant with HTML5. Most modern browsers support videos having Ogg containers with Vorbis audio codec and Theora video codec or WebM containers with Vorbis audio codec and VP8 video codec. Users must be able to upload videos in a format of their choice like MPEG, DivX or AVC/H.264. Thus, the platform must perform a conversion from the input user format to a NextShare compliant format. P2P-Tube converts all videos to Ogg with Vorbis and Theora in order to be compatible with most modern browsers and to make video assets playable with any of the two NextShare plugins. However this can be changed from the platform configuration.
+NextSharePC can play any video format as long as VLC supports it and SwarmPlayer supports only video containers and codecs which are compliant with HTML5. Most modern browsers support videos having Ogg containers with Vorbis audio compression and Theora video compression or WebM containers with Vorbis audio compression and VP8 video compression. Users must be able to upload videos in a format of their choice like MPEG, DivX or AVC/H.264. Thus, the platform must perform a conversion, also known as \textit{transcoding}, from an input user format to a NextShare compliant format. P2P-Tube converts all video assets to videos with Ogg containers and Vorbis+Theora compression in order to be compatible with most modern browsers and to make video assets playable with any of the two NextShare plugins. However this can be changed from the platform configuration.
+
+Besides video transcoding, several other operations must be performed in order to make new video assets available on the platform. This is done by a \textbf{Content Ingestion Server} (\textbf{CIS}) which is typically located on a different machine.
-Besides video conversion, several other operations must be performed in order to make the new video available on the platform. All this is done by a \textbf{Content Ingestion Server} (\textbf{CIS}) which is typically located on a different machine.
+An user uploads a video file, referred here as \textit{raw video file}, to a web server and may also provide additional information about it, like title, description, tags etc.. When video upload is finished the process of content ingestion is started by the web server by sending a content ingestion request to a CIS. This is done directly, if the system has a single CIS, or indirectly via a CIS Load Balancer (CIS-LB), which forwards the request to one CIS from a pool of several CIS machines. By doing load balancing, the indirect request method provides scalability because more content ingestion jobs can run in parallel, allowing more users to add content simultaneously to the platform. By these means availability is also increased, the system becoming fault tolerant in case of CIS failures.
+
+Note that sending content ingestion request is transparent for the web server. There is no difference for it if this is done directly to a CIS or indirectly through a CIS-LB. This simplifies implementation of the web server and hides the abstraction of content ingestion from it.
\begin{figure}[h]
\begin{center}
\label{fig:content-ingestion}
\end{figure}
-An user uploads a video file (referred here as \textit{raw video file}) to a the web server and may also provide additional information about it, like title, description, tags etc.. When the video upload finishes the process of content ingestion is started by the Web Server by sending a content ingestion request to a CIS. This can be done directly if in the system is a single CIS or indirectly via a CIS Load Balancer (CIS-LB), which forwards the request to one CIS from a pool of CIS machines. By doing load balancing, the indirect request method provides more scalability by offering the opportunity of running more content ingestion jobs in parallel, allowing more users to add content simultaneously. This also increases availability by making the system fault tolerant in case of CIS faults.
-
-CIS operation through a CIS Load Balancer is described in Figure \ref{fig:content-ingestion}. Until the end of this subsection, the numbers in brackets refer to the arrows from the figure. The web server sends a \textbf{content ingestion request} to CIS-LB (1), which must choose the least loaded CIS from the system. To do this it queries each CIS for its load by sending a \textbf{get load request} (2) and establishes by processing the answer which one is the least loaded. The content ingestion request previously received is then forwarded to the least loaded CIS (3), informing it about the new video file that has been uploaded. The message also contains information about one or more formats in which the video must be converted referred here as \textit{destination formats}. This information includes parameters for each format like image resolution, frame rate, audio quality etc.. The least loaded CIS which received the request executes a job on the raw video file uploaded by doing the following operations:
+CIS operation through a CIS Load Balancer is described in Figure \ref{fig:content-ingestion}. Until the end of this subsection, the numbers in brackets refer to arrows from the figure. The web server sends a \textbf{content ingestion request} to CIS-LB (1), which must choose the least loaded CIS from the system. To do this it queries each CIS for its load by sending a \textbf{get load request} (2) and establishes by processing the answer which one is the least loaded. The content ingestion request previously received is then forwarded to the least loaded CIS (3), informing it about the new video file that has been uploaded. The message also contains information about one or more formats in which the video must be converted referred here as \textit{destination formats}. This information includes parameters for each format like image resolution, frame rate, audio quality etc.. The least loaded CIS which received the request executes a job on the uploaded raw video file by doing the following operations:
\begin{itemize}
\item \textbf{Transfers locally} (4) the raw video file from the web server.
\item \textbf{Creates torrent files} (7) (\textit{.tstream}) for transcoded video files.
\item \textbf{Seeds} (8) the transcoded video files by using the torrent files previously created.
\item \textbf{Transfers back} (9) torrent files and image thumbnails to the web server.
- \item Notifies the Web Server of the \textbf{job completion} (10) so that it can make the video accessible to the users.
+ \item Notifies the Web Server about the \textbf{job completion} (10) so that it can make the video accessible to the users.
\end{itemize}
-In the end of the content ingestion process all other Content Ingestion Servers from the system must be aware of the new torrent files in order to seed them. This can be achieved by putting all torrent file in a distributed files system.
\ No newline at end of file
+In the end of the content ingestion process all other Content Ingestion Servers from the system must be aware of the new torrent files in order to seed them. First of all, the new torrent files need to be locally accessible to them, so all torrent files can be shared by CIS machines on a \textit{distributed file system} such as NFS, LustreFS or GlusterFS. The overhead of using shared storage between CIS machines shouldn't affect performance, even if NFS is used, which serves files through a single machine. This is because torrent files are very small so concurrently accessing more files on the shared file system can affect performance in a negligible manner.
+
+When the web server is notified about the job completion (10) a seed request must arrive to each CIS so that they can start seeding the new torrent files available in the distributed file system. This request is originated from the web server and there is no need to do this on a single CIS system, because this is already done in step 8 from Figure \ref{fig:content-ingestion}, so the web server can be configured to skip this redundant step from its configuration. In a multiple CIS system this step is mandatory for ensuring high availability of the new video assets. Thus, a \textbf{seed request} is sent from web server to CIS-LB, which forwards this message to all CIS from its jurisdiction. Upon reception they will start seeding requested torrent files which are located in the distributed file system.
\ No newline at end of file
-% vim: set tw=78 sts=2 sw=2 ts=8 aw et ai:
-%
-% Owner: Călin-Andrei Burloiu
-%
-\subsection{Programming Languages, Frameworks}
-\label{subsec:langs-and-frameworks}
+One of the basic design principles of our platform was to publish it as open-source encouraging community contributions. To facilitate this we have chosen popular programming languages, frameworks and technologies in order to better attract open-source communities around our platform. These decisions along with implementation details are given in this section. Information found here accompanied with comments from the source code should hopefully provide enough information to a new contributor to this project.
-P2P-Tube server side application is implemented in PHP using CodeIgniter \cite{code-igniter} open-source framework. This increases opportunity to integrate this project in an open-source community, because both the programming language and the framework are very popular.
+Our implementation was designed to work on \textbf{GNU/Linux} distributions, so our back end dependencies, such as FFmpeg, has been chosen based on this. However we tried as much as possible to choose them such that they also run on other operating systems, such as Windows. Thus, porting P2P-Tube for them should be easy.
-On the client side, besides HTML and CSS, a very popular JavaScript framework is used, that is jQuery \cite{jquery}. As mentioned before at least one of the NextShare plugins need to be installed in order to play videos. SwarmPlayer requires HTML5 capabilities for video tag support and compatibility with Ogg video container and Vorbis + Theora codecs. A jQueryUI \cite{jqueryui} widget was developed which is used to control video playing with features like play, pause and stop buttons, time progress slider, volume control etc..
+\subsection{Web Application Front End}
+\label{subsec:front-end}
-Content Ingestion Server is written in Python, mainly because of the need of using threads. A master thread takes the role of a producer by receiving requests from clients and submitting them to a job queue. The consumer is a worker thread which gets jobs from the queue and executes them, using a first-come first-server servicing policy. If the queue is empty the worker waits without blocking until a new job is available. Client requests are received by the master with the aid of Web.py, a lightweight web framework \cite{webpy}.
+The front end of P2P-Tube provides an interface with the users of the platform. Today more and more applications are migrating from a classical desktop interface to a web interface. For instance, applications like Microsoft Office started to appear on the cloud, providing a web interface; one popular example is Google Docs. Besides this new trend, Next-Share content delivery platform was successfully implemented in browsers by P2P-Next. So implementing P2P-Tube as a web platform came as a natural choice.
+
+Because almost any hosting provider offers a standard LAMP stack, we decided to cope we such a configuration. LAMP stands for Linux, Apache, MySQL, PHP, the technologies which is based on. We tested P2P-Tube on an Apache web server, but it should work well on others.
+
+P2P-Tube server side application is implemented in \textbf{PHP} which is one of the most popular programming languages for this purpose. To facilitate development we decided to use \textbf{CodeIgniter} \cite{code-igniter} open-source framework. Thanks to this framework the platform uses a \textit{MVC (Model-View-Controller)} design pattern, which separates the user interface part (View), from the data stored (Model) under the rule of the Controller.
+
+Our implementation uses a \textbf{MySQL} database, but CodeIgniter abstracts the interface with the database. So porting our platform for a different SQL technology such as PostgreSQL should be easy.
+
+As a web application, on the client side P2P-Tube uses HTML, CSS and JavaScript. As stated before SwarmPlayer only works with browsers that support HTML5. We decided to use jQuery \cite{jquery} as a JavaScript framework to ease development on the client side programming. We developed a jQueryUI \cite{jqueryui} widget which is used to control video playing with features like play, pause and stop buttons, time progress slider, volume control etc.. This widget is detailed in Subsection \ref{subsec:video-widget}.
+
+\subsection{Video Widget}
+\label{subsec:video-widget}
-\subsection{Interoperability}
-\label{subsec:interoperability}
+TODO
-The web server needs to communicate with Content Ingestion Server, whether or not through a CIS Load Balancer (CIS-LB), by sending a content ingestion request. By using a web service for this communication the overhead is reduced. Sending HTTP requests from PHP web server application is much more easy then creating a custom new protocol. The same applies at the other communication point, the Python server, where requests are received by Web.py framework, which makes HTTP methods processing extremely easy.
+\subsection{Content Ingestion Back End}
+\label{subsec:back-end}
-If another framework decides to use our solution based on CIS, interoperability is simplified by using web service interfaces, no matter what programming language is used for web server application.
+TODO
+
+\subsection{Front End and Back End Communication}
+\label{subsec:communication}
+
+The web server needs to communicate with Content Ingestion Server, whether or not through a CIS Load Balancer (CIS-LB), by sending a content ingestion request. By using a web service for this communication the implementation overhead is reduced. Sending HTTP requests from the PHP web server application is much more easy then creating a custom new protocol. The same applies at the other communication point, the Python server, where requests are received by Web.py framework, which makes HTTP methods processing extremely easy.
+
+If another framework decides to use our solution based on CIS machines, interoperability is simplified by using web service interfaces, no matter what programming language is used for web server application.
+
+\subsection{Programming Languages, Frameworks}
+\label{subsec:langs-and-frameworks}
+
+Content Ingestion Server is written in Python, mainly because of the need of using threads. A master thread takes the role of a producer by receiving requests from clients and submitting them to a job queue. The consumer is a worker thread which gets jobs from the queue and executes them, using a first-come first-server servicing policy. If the queue is empty the worker waits without blocking until a new job is available. Client requests are received by the master with the aid of Web.py, a lightweight web framework \cite{webpy}.
\subsection{The Choice for the Web Service}
\label{subsec:securing-cis}
projects / living-lab-site.git / commitdiff