ifkad articles

Spatio-Temporal Types of Data in Big Data Paradigm

Mladen Amović, Miro Govedarica, Vladimir Pajić, Slavko Vasiljević

The development of technologies contributed to the exponential increase in the volume of the collected spatio-temporal data. It was determined that parallel processing of large series of spatio-temporal data can contribute to their analysis takes only a few seconds instead of hours. Big data applications require a combination of different process techniques, data sources and formats of storage. Spark SQL allows programmers great advantages in relational processing such as calling complex analytic libraries in Spark (eg. Machine learning). Also provides a general framework for the transformation of the structure, which we use to perform the analysis, planning and code generation in real time, expanding with new data sources, including data such as JSON and “smart” data warehouse over which is possible filtering (such as HBase) with user defined functions and user-defined types and domains such as machine learning. Model for managing large volumes of spatio-temporal data is implemented in Apache Spark platform for storing and processing large sets of data. The algorithms for processing spatio-temporal data are defined according to the rules of Spark SQL programming model and relational operations on dataframes (specialized system of data frames) using domain specific language (domain – specific – language → DSL). Data are stored on external storage systems that support new data types. Implementation of algorithm is performed in the programming language Scala. There is relatively small number of research in the field spatio-temporal Big Data and there are only several publications related to our research. Our model is based on the Spark which currently represents de-facto standard for Big Data processing. We utilise all advantages provided by Spark, such as user-defined types, user-defined functions, and DSL in order to support new spatio-temporal data types. The functionality of our model is accessible through SQL and DSL constructs and therefore available to wide spectrum of users and not only programmers. Amount of spatio-temporal data grows continuously at very fast pace. In order to use and maximally utilise the potential of such amounts of data, new solutions for storage, distribution, indexing, processing, and presentation, are necessary. The model we proposed is based on well established platforms for Big Data, such as Spark, Hadoop, and HBase, which inherently provides the solutions for the most of problems mentioned. We introduced new data types in those platforms in order to provide support for spatio-temporal data which were defined in accordance with existing OGC and ISO standards.

IN: Proceedings IFKAD 2016 – Towards a New Architecture of Knowledge: Big Data, Culture and Creativity
PP: 466-479