{"id":319223,"date":"2021-06-28T15:27:00","date_gmt":"2021-06-28T12:27:00","guid":{"rendered":"https:\/\/inform.com.de\/?p=319223"},"modified":"2021-06-26T18:06:42","modified_gmt":"2021-06-26T15:06:42","slug":"what-is-big-date-simple-answer-for-everyone-big-data-big-responsibility-big-stress-and-money","status":"publish","type":"post","link":"https:\/\/inform.com.de\/en\/what-is-big-date-simple-answer-for-everyone-big-data-big-responsibility-big-stress-and-money\/","title":{"rendered":"What is Big Date? Simple answer for everyone. Big Data – big responsibility, big stress and money"},"content":{"rendered":"

An excursion into history and statistics<\/h2>\n

The phrase “big data” appeared in 2008 with the light hand of Clifford Lynch. In a special issue of the journal Nature, the expert called the explosive growth of information flows – big data. In it, he took any arrays of heterogeneous data over 150 GB per day.<\/p>\n

From the statistical calculations of analytical agencies in 2005 the world operated with 4-5 exabytes of information (4-5 billion gigabytes), after 5 years the volume of big data increased to 0.19 zettabytes (1 ZB = 1024 EB). In 2012, the numbers increased to 1.8 ZB, and in 2015 – to 7 ZB. Experts predict that by 2020, big data systems will operate 42-45 zettabytes of information.<\/p>\n

Until 2011, big data technologies were considered only as scientific analysis and had no practical solution. However, the amount of data grew exponentially, and the problem of huge amounts of unstructured and heterogeneous information became relevant in early 2012. The surge in interest in big data is clearly visible in Google Trends.<\/p>\n

Digital business mastodons – Microsoft, IBM, Oracle, EMC and others – have joined the development of the new direction. Since 2014, big data has been studied at universities, implemented in applied sciences – engineering, physics, sociology.<\/p>\n

What is Big Data?<\/h2>\n

Big data is an information processing technology that exceeds hundreds of terabytes and is growing exponentially over time.<\/p>\n

Such data is so large and complex that none of the traditional data management tools can store or process it efficiently. A person is not able to analyze this volume. For this, special algorithms have been developed that, after analyzing big data, give a person understandable results.<\/p>\n

Big Data includes petabytes (1,024 terabytes) or exabytes (1,024 petabytes) of information that make up billions or trillions of records by millions of people, all from different sources (Internet, sales, contact center, social media, mobile devices). As a rule, information is poorly structured and often incomplete and unavailable.<\/p>\n

What companies are doing big data<\/h2>\n

Cellular operators and search engines were the first to work with big data, or with “big data”. The search engines were getting more and more queries, and the text was heavier than numbers. A paragraph of text takes more time to work with than a financial transaction. The user expects the search engine to process the request in a split second – it is unacceptable that it work even for half a minute. Therefore, search engines were the first to start working with parallelization when working with data.<\/p>\n

A little later, various financial organizations and retail joined in. Their transactions themselves are not so large, but big data appears due to the fact that there are a lot of transactions.<\/p>\n

The amount of data is growing in general for everyone. For example, banks had a lot of data before, but they did not always require operating principles, as with large ones. Then banks began to work more with customer data. They began to come up with more flexible deposits, loans, different tariffs, and began to analyze transactions more closely. This already required quick ways of working.<\/p>\n

Now banks want to analyze not only internal information, but also third-party information. They want to receive big data from the same retail, they want to know what a person spends money on. Based on this information, they try to make commercial offers.<\/p>\n

Now all information is being linked together. Retail, banks, telecom operators and even search engines are now all interested in each other's data.<\/p>\n

What should be a big data specialist<\/h2>\n

Since the data is located on a cluster of servers, a more complex infrastructure is used to process it. This puts a lot of stress on the person who works with it – the system must be very reliable.<\/p>\n

Making a single server reliable is easy. But when there are several of them, the probability of a fall increases in proportion to the number, and the responsibility of the data engineer who works with this data also grows.<\/p>\n

The big data analyst must understand that he can always receive incomplete or even incorrect data. He wrote the program, trusted its results, and then found out that due to the fall of one server out of a thousand, part of the data was turned off, and all the conclusions were incorrect.<\/p>\n

Take text search, for example. Let's say all words are arranged in alphabetical order on several servers (to put it very simply and conditionally). And then one of them turned off, all the words with the letter “K” disappeared. The search stopped giving out the word “Cinema”. Then all the news disappears, and the analyst makes the false conclusion that people are no longer interested in cinemas.<\/p>\n

Therefore, a big data scientist must know the principles of work from the lowest levels – servers, ecosystems, task schedulers – to the highest-level programs – machine learning libraries, statistical analysis, and more. He must understand the principles of hardware, computer equipment and everything that is configured on top of it.<\/p>\n

For the rest, you need to know everything the same as when working with small data. You need mathematics, you need to be able to program and be especially familiar with distributed computing algorithms, to be able to apply them to the usual principles of working with data and machine learning.<\/p>\n

What big data tools and technologies are used<\/h2>\n

Since the data is stored on a cluster, a special infrastructure is needed to work with it. The most popular ecosystem is Hadoop. A lot of different systems can run in it: special libraries, schedulers, machine learning tools, and much more. But first of all, this system is needed to analyze large amounts of data through distributed computing.<\/p>\n

For example, we are looking for the most popular tweet across a thousand servers. On one server, we would just make a table and that's it. Here we can drag all the data to ourselves and recalculate. But this is not correct, because it takes a very long time.<\/p>\n

Therefore, there is Hadoop with Map Reduce paradigms and Spark framework. Instead of pulling data to themselves, they send program sections to this data. The work is going on in parallel, in a thousand threads. Then you get a sample of thousands of servers, on the basis of which you can select the most popular tweet.<\/p>\n

Map Reduce is an older paradigm, Spark is newer. With its help, they get data from clusters, and build machine learning models in it.<\/p>\n

Big data market in Russia<\/h2>\n

\"What<\/a><\/p>\n

The whole world, including Russia, uses Big Data technology in the banking sector, communications services and retail. Experts believe that in the future, the technology will be used by the transport industry, oil and gas and food industries, as well as energy.<\/p>\n

IDC analysts have recognized Russia as the largest regional market for BDA. Revenue is estimated to be close to $ 1.4 billion this year and will account for 40% of total investments in big data and business intelligence applications.<\/p>\n

What professions are there in the field of big data<\/h2>\n

The two main professions are analysts and data engineers.<\/p>\n

The analyst primarily works with information. He is interested in tabular data, he is engaged in models. His responsibilities include aggregating, cleaning, augmenting and visualizing data. That is, the big data analyst is the link between raw information and the business.<\/p>\n

The analyst has two main lines of work. First, he can transform the information received, draw conclusions and present it in an understandable form.<\/p>\n

Second, analysts develop applications that will work and produce results automatically. For example, make a forecast for the stock market every day.<\/p>\n

The date engineer is a lower-level specialty. This is the person who must ensure the storage, processing and delivery of information to the analyst. But where there is supply and cleaning, their responsibilities can overlap.<\/p>\n

Bigdata engineer gets all the hard work. If the systems have failed, or one of the servers has disappeared from the cluster, it connects. This is a very demanding and stressful job. The system can shut down on weekends and after hours, and the engineer must take action promptly.<\/p>\n

These are two main professions, but there are others as well. They appear when parallel computing algorithms are added to tasks related to artificial intelligence. For example, an NLP engineer. This is a programmer who deals with natural language processing, especially in cases where it is necessary not only to find words, but to grasp the meaning of the text. Such engineers write programs for chat bots and dialog systems, voice assistants and automated call centers.<\/p>\n

There are situations when you need to classify billions of images, make moderation, filter out unnecessary things and find similar ones. These professions overlap more with computer vision.<\/p>\n

How long does training take<\/h2>\n

We have been training for a year and a half. They are split into six quarters. Some focus on programming, others on working with databases, and still others on mathematics.<\/p>\n

Unlike, for example, the AI \u200b\u200bfaculty, there is less mathematics here. There is no such strong emphasis on calculus and linear algebra. Knowledge of distributed computing algorithms is needed more than the principles of calculus.<\/p>\n

But a year and a half is enough for real work with big data processing only if a person had experience working with ordinary data and in general in IT. The rest of the students after graduating from the faculty are encouraged to work with small and medium data. Only then can a specialist be allowed to work with large ones. After training, it is worth working as a data scientist – to apply machine learning on different amounts of data.<\/p>\n

When a person gets a job in a large company – even if he had experience – most often he will not be admitted to large amounts of data right away, because the cost of error is much higher there. Errors in algorithms may not be detected immediately, and this will lead to large losses.<\/p>\n

How does Big-Data technology work?<\/h2>\n

\"What<\/a><\/p>\n

Users of the social network Facebook upload photos, videos and perform actions every day for hundreds of terabytes. No matter how many people participate in the development, they will not cope with the constant flow of information. To further develop the service and make sites more comfortable – to implement smart content recommendations, display ads relevant to the user, hundreds of thousands of terabytes are passed through the algorithm and receive structured and understandable information.<\/p>\n

Comparing a huge amount of information, relationships are found in it. These relationships with a certain probability can predict the future. Artificial intelligence helps to find and analyze a person.<\/p>\n

The neural network scans thousands of photos, videos, comments – those very hundreds of terabytes of big data and gives the result: how many satisfied customers leave the store, whether there will be a traffic jam in the next few hours, what discussions are popular on the social network, and much more.<\/p>\n

Methods for working with big data:<\/strong><\/p>\n