Big data refers to a collection of data that cannot be captured, managed, and processed by conventional software tools within a certain time frame. It is a massive amount of decision-making, insight, and process optimization capabilities that require new processing models. High growth rates and diverse information assets.
In the Big Data Era written by Victor Meyer-Schonberg and Kenneth Cooke, big data means that instead of random analysis (sampling survey), all data is used for analysis. The 5V characteristics of big data (proposed by IBM): Volume, Velocity, Variety, Value, Veracity.
This definition is given to Gartner, the "big data" research organization. “Big Data†is an information asset that requires a new processing model to have greater decision making, insight and process optimization capabilities to accommodate massive, high growth rates and diversification.
Undoubtedly, all the people in the world who are concerned about developing technology are aware of the potential value of “big data†for business commerce. The purpose is to solve the pain caused by the growth of various business data in the process of enterprise development.
The reality is that many problems hinder the development and practical application of big data technology.
Because of a successful technology, some measure of the standard is required. Now we can measure big data technology with a few basic elements, which are -- stream processing, parallelism, digest indexing, and visualization.
1, stream processing
Along with the pace of business development and the complexity of business processes, our focus is increasingly on "data streams" rather than "data sets."
Decision makers are interested in keeping track of the lifeblood of their organization and getting real-time results. What they need is an architecture that can handle the flow of data that occurs at any time. Current database technology is not suitable for data stream processing.
For example, calculating the average of a set of data can be done using a traditional script. But for the calculation of the moving data average, there are more efficient algorithms, whether it is arriving, growing or one unit after another. If you want to build a data warehouse and perform arbitrary data analysis and statistics, open source products R or commercial products like SAS can be implemented. But what you want to create is a data flow statistics set, which gradually adds or removes data blocks, performs moving average calculations, and the database does not exist or is not mature.
The ecosystem surrounding the data stream is underdeveloped. In other words, if you are negotiating a big data project with a vendor, you must know if data stream processing is important to your project and if the vendor is capable of providing it.
2, parallelization
There are many definitions of big data, and the following is relatively useful. The "small data" situation is similar to the desktop environment, the disk storage capacity is between 1GB and 10GB, the "medium data" data volume is between 100GB and 1TB, and the "big data" distributed storage is on multiple machines, including 1TB to multiple PB data.
If you work in a distributed data environment and want to process data in a short amount of time, this requires distributed processing.
Parallel processing stands out in distributed data, and Hadoop is a well-known example of distributed/parallel processing. Hadoop includes a large distributed file system that supports distributed/parallel queries.
3. Summary index
A summary index is a process of creating a precomputed summary of data to speed up the execution of the query. The problem with the summary index is that you have to plan for the query to be executed, so it has limitations.
The data is growing rapidly, and the requirements for the digest index will not stop. Whether it is long-term consideration or short-term, the supplier must have a certain strategy for the development of the digest index.
4, data visualization
Visual chemical industry has two major categories.
Exploratory visual description tools help decision makers and analysts explore the connections between different data, a visual insight. Similar tools are Tableau, TIBCO, and QlikView, which are a class.
Narrative visualization tools are designed to explore data in a unique way. For example, if you want to visually view a company's sales performance in a time series in a time series, the visualization format is pre-created. The data is presented monthly by region and sorted according to predefined formulas. The supplier PercepTIve Pixel falls into this category.
What are big data technologies?1, cross-granular calculation (In-DatabaseCompuTIng)
Z-Suite supports a variety of common summaries and supports almost all professional statistical functions. Thanks to the cross-granular computing technology, the Z-Suite data analysis engine will find the optimal computing solution, and then move all the expensive and expensive calculations to the data storage directly. We call it the library. Calculation (In-Database). This technology greatly reduces data movement, reduces communication burden, and ensures high-performance data analysis.
2, parallel computing (MPP CompuTIng)
Z-Suite is a business intelligence platform based on MPP architecture. It can distribute calculations to multiple computing nodes and then summarize the calculation results at specified nodes. Z-Suite is able to take advantage of a variety of computing and storage resources, whether it is a server or a normal PC, she does not have strict requirements on network conditions. As a horizontally extended big data platform, Z-Suite can fully utilize the computing power of each node to easily achieve a second-level response for TB/PB-level data analysis.
3, column storage (Column-Based)
Z-Suite is column stored. Data storage based on column storage, without reading irrelevant data, can reduce read and write overhead, while improving I/O efficiency, thereby greatly improving query performance. In addition, column storage can better compress data, generally between 5 and 10 times the compression ratio, so that the data footprint is reduced to 1/5 to 1/10 of the traditional storage. Good data compression technology saves storage device and memory overhead, but greatly improves computing performance.
4, memory calculation
Thanks to column storage technology and parallel computing technology, Z-Suite can greatly compress data while taking advantage of the computing power and memory capacity of multiple nodes. In general, memory access speeds are hundreds or even thousands of times faster than disk access speeds. Through memory calculations, the CPU reads data directly from memory rather than disk and computes the data. Memory computing is an acceleration of traditional data processing and a key application technology for big data analysis.
Precautions
The strategic significance of big data technology is not to master huge data information, but to specialize in these meaningful data. In other words, if big data is likened to an industry, then the key to profitability in this industry is to increase the “processing power†of the data and “add value†of the data through “processingâ€.
Car Screen Protector,Car Center Console Screen Protector,Car Touchscreen Screen Protector
Shenzhen Jianjiantong Technology Co., Ltd. , https://www.jjttpucuttingplotter.com