Big Data

Big data is all about the storage, processing and analysis of enormous amounts of data using special hardware and software. In the corporate context in particular, this creates numerous opportunities to optimize existing processes, tap into undiscovered market potential and adopt a more target-group-oriented approach. Big data will unquestionably remain one of the key topics of the 21st century – a fact made clear by the following Statista forecast: While the amount of data generated worldwide in 2018 was still 33 zettabytes (1,000,000,000,000,000,000 bytes), experts forecast an increase to 175 zettabytes(1) by the year 2025.

Table of contents

What five characteristics best describe big data?
Why is big data more relevant than ever for companies today?
How can companies successfully introduce the use of big data?
In what areas of business is big data used?
What competitive advantages can be achieved?
What challenges does the use of big data pose?
What new job profiles is big data creating?
How Brunel can support you in your next big data project

Social networks, mobile devices, the Internet of Things: Data is generated from an incredibly broad spectrum of sources, and the volume of data is unimaginably large. Facebook generates 50,000 clicks per second, while YouTube registers 72 hours of new video material per minute. In other words, 2.5 trillion bytes are generated worldwide every day. Data that used to be considered unusable can nowadays yield numerous competitive advantages – reason enough to take a closer look at big data.

What five characteristics best describe big data?

Volume: This refers to the enormous amount of data that is a distinguishing feature of big data. Almost every transaction, be it in the commercial or private realm, generates data – from bank transfers to development and production processes. The world's largest data producer is the manufacturing industry, which accounts for around 3.6 zettabytes (2).

Velocity: Today, data is not only collected but must also be processed at a comparatively high speed. This creates a need for technical solutions for real-time processing, such as operating systems, sensors and smart metering. The velocity of data dissemination has also increased considerably in recent years.

Variety: Data is available in different formats, with distinctions drawn between structured, semi-structured and unstructured data. Whereas structured data can be represented in relational databases in the form of rows and columns, unstructured data is available as images, text files, animations and even audio sequences in the form of numeric codes. The latter make up almost 80 percent of all data in companies. Big data and machine learning can be used to analyze and further utilize this data.

Veracity (truthfulness of data quality): High-quality data is a prerequisite for reaping the benefits of big data. Only if it can be ensured that the data is authentic, complete and unambiguous can big data be of real benefit to companies.

Value: The fifth characteristic of big data is its value to businesses. Valuable information about processes or customers can be collected or predicted and can be a real asset to business strategy.

Why is big data more relevant than ever for companies today?

How can companies successfully introduce the use of big data?

1. Define your strategy

Companies that decide to exploit the potential of big data are first faced with the challenge of defining a strategy for its introduction. In particular, this involves determining how data is to be collected, managed and used within the organization. The following questions can be extremely useful in this context:

• What goals do you want to achieve with big data?
• Is everyone involved aware of the value of the data, or do they still need to be convinced?
• Does your company have the right big data capabilities? Do specialists need to be recruited or even an external service provider brought in?
• Is it necessary to adapt the existing IT infrastructure with regard to existing data sources and subsequent data analysis?
• How should data be collected, stored and managed? (If this is already happening, an inventory should be made and potential for improvement identified).
• Does additional software need to be purchased?
• Which stakeholders within the company need to be considered, trained and connected to the new system landscapes when using big data?
• What other data-specific aspects, such as data governance and data protection, need to be taken into account?

2. Analyze big data sources

Big data stands or falls with the quality of the data. Therefore, it is crucial for companies to identify which data sources and data streams are available. Data sources and streams can differ from company to company and can include the following (for example):

Social networks: Facebook, Instagram, XING and LinkedIn serve as communication and interaction platforms for millions of people worldwide. Gigantic amounts of data are generated via these networks every day.

Mobile devices: Data from mobile devices provides information about the whereabouts of their users. This offers numerous benefits, especially for location-driven marketing.

Multimedia data: Multimedia data now accounts for the majority of data on the Internet. YouTube alone adds 75 hours of new video material every minute. Due to the wealth of data that videos contain, multimedia data is particularly valuable.

Text files: Text files also form a valuable data resource. It is now possible to search millions of documents in real time.

3. Manage your data

Once the data sources have been identified, it is necessary to consider how they can be accessed and how the data should be stored and managed. In this context, it is important to ask whether additional software systems need to be purchased that are fast enough, powerful enough and flexible enough to access the necessary data. Adequate data quality is closely connected to this issue and should be ensured to provide users with the greatest possible benefits. If the data comes from different sources and is available in different formats, it should be prepared in a uniform way to avoid compatibility problems. Finally, data governance and master data management should be taken into account. The former includes the rules and regulations to ensure that data storage and processing comply with the company's business strategy. In the context of master data management, a unified corporate decision must be taken on how data is stored and monitored in the company.

4. Choose the right analysis techniques

Which analytical methods are used depends heavily on the purpose of analysis. Common analytical methods include, for example, text mining, in which larger amounts of text are examined for information that can be analyzed. Data mining, on the other hand, examines incoming data sets to find correlations, commonalities and regular patterns so that relevant information can be gained for the company. Yet another element is what is known as process mining, in which complex business and/or production processes are examined on the basis of the data they generate and optimized if necessary. The first points of contact with the topic of artificial intelligence are in machine learning, in which machines can automatically identify correlations based on data and then, on this basis, reproduce data that can be used for strategic corporate decisions, for example.

In what areas of business is big data used?

What competitive advantages can be achieved?

What challenges does the use of big data pose?

Technological challenges: At the technological level, there is still a lot of uncertainty about which software systems and tools are the right ones for a given use case. From Spark to Hadoop MapReduce to Cassandra and Hbase, the market for big data technologies is large indeed. It is therefore a real challenge for the layperson to decide on the right systems and find the right experts for a particular application.

Business challenges: Introducing big data technologies can be expensive, especially if new software and hardware have to be purchased and the appropriate specialist staff recruited. That said, how high the costs are in individual cases depends to a large extent on the goals and requirements of the company. Companies that have strict security requirements are well served by on-premises solutions but will then have to factor in costs for hardware and new skilled staff. Companies that attach great importance to flexibility are well advised to opt for cloud solutions but will then have to invest in cloud services and, again, suitably skilled staff.

Operational challenges: While high data quality is the be-all and end-all for big data analytics, ensuring high-quality data still constitutes a significant hurdle for many companies. Often, data comes from different sources and in different formats, so harmonizing all the data can be a problem. In addition, the data must be reliable if meaningful analysis is to be possible at all. One issue that should not be neglected here is possible security loopholes, which are often overlooked in big data analytics. Lastly, valuable conclusions for the company’s business must be derived from the results of analysis, which calls for big data experts and their special expertise.

What new job profiles is big data creating?

Due to the enormous amounts of data involved, many companies are investing in the development of new tools and systems to optimize data storage, management and analysis. This development brings with it new job profiles in the IT environment, such as big data scientists and data artists, both of whom specialize in managing huge amounts of data and all the associated activities and technologies. Already established but still comparatively new is the domain of data protection officers, demand for whom is likely to increase significantly in the future due to big data.

To find the right experts for your project, we start by acquiring a comprehensive overview of your project. We want to know what tools, experience, qualifications and language skills our experts should have. In this way, we can ensure that our specialists develop big data solutions that meet your individual quality requirements.

Subsequently, our talent scouts and account managers go in search of suitable candidates via various channels and networks. A proactive and forward-looking approach to candidates as well as our global network of technical and IT experts are of great benefit to us in this context. Would you like to put our recruiting expertise to the test? Go here to find out more.

What new job profiles is big data creating?

Due to the enormous amounts of data involved, many companies are investing in the development of new tools and systems to optimize data storage, management and analysis. This development brings with it new job profiles in the IT environment, such as big data scientists and data artists, both of whom specialize in managing huge amounts of data and all the associated activities and technologies. Already established but still comparatively new is the domain of data protection officers, demand for whom is likely to increase significantly in the future due to big data.

To find the right experts for your project, we start by acquiring a comprehensive overview of your project. We want to know what tools, experience, qualifications and language skills our experts should have. In this way, we can ensure that our specialists develop big data solutions that meet your individual quality requirements.

Subsequently, our talent scouts and account managers go in search of suitable candidates via various channels and networks. A proactive and forward-looking approach to candidates as well as our global network of technical and IT experts are of great benefit to us in this context. Would you like to put our recruiting expertise to the test? Go here to find out more.

How Brunel can support you in your next big data project?

Benefit from our experts’ in-depth knowledge of big data technologies, data modeling and data visualization. Combining this with our cross-industry expertise, we can take on almost any project in the field of big data – professionally, flexibly and in line with each customer’s needs.

To find the right experts for your project, we start by acquiring a comprehensive overview of your project. We want to know what tools, experience, qualifications and language skills our experts should have. In this way, we can ensure that our specialists develop big data solutions that meet your individual quality requirements.

Subsequently, our talent scouts and account managers go in search of suitable candidates via various channels and networks. A proactive and forward-looking approach to candidates as well as our global network of technical and IT experts are of great benefit to us in this context. Would you like to put our recruiting expertise to the test? Go here to find out more.