Xebia Articles

The monitoring of compliance with the GDPR will start in just over a month. What does that mean for your work as a developer or data scientist? "You have to continuously ask yourself what data you really need, and how you can minimize the impact of its use."

One of the defining aspects of a public blockchain is that anyone can add information to it. However, before it is added, the information must be verified. All cooperating nodes in the network must agree on the definitions of valid, and invalid, information. For example, in the case of Bitcoin, there must be a consensus on what defines a legitimate transaction. When the new information is deemed valid by all participating nodes in the network, then the consensus is reached. This article explores how a decentralized network could achieve consensus through two different algorithms: proof-of-work and proof-of-stake.

Today, most companies still rely on either traditional databases or data warehouses to generate the reports their management needs to make informed decisions. Those solutions have gotten us pretty far, but they also have some serious drawbacks. For one, loading all that data into the same place and then generating reports can take hours - and that’s only if everything works right the first time around! If something goes wrong during the process (and something often does) it can take even longer.

“There’s lots of value in data analytics. But when the low-hanging fruit in a dataset is gone, it becomes harder to extract value from the data.” It is tempting for organizations to find biased answers and draw faulty conclusions, like mixing causation with correlation. In a recent presentation at the Meetup Business Experimentation, Adam Kelleher, lead data scientist at Buzzfeed, emphasized that this is not without risk.

Putting predictive models into production demands a streamlined workflow, and highly skilled data scientists and engineers. For the past few years, organizations have focused on developing strong proof of concepts and initial use cases. Now they must successfully introduce data-driven concepts into their daily operations. This year, data-driven companies will make productionizing data projects the priority and focus. Productionizing will increase exposure in other parts of the company, so data science teams will turn to proven methods, like microservices and APIs, to streamline their process. Data governance will become a standard part of any data project. Big Data technology has matured and is mostly enterprise-ready, increasing the pressure on and demand for data scientists. Major tech players like Google, LinkedIn, and Facebook will continue to open-source their product innovations in an early stage. In 2017, the gap between textbook data scientists and seasoned practitioners will widen further. Companies that retain, foster, and attract people who can apply data science in production will gain a significant competitive advantage.

In Data Science, software quality often is an issue that prevents models to hit production. Issues like no automated data pipelines (including how to make the results available to the outside world), bad quality of code, or not enough attention to non functional requirements (like performance) are showstoppers for applied data science.

How can you successfully bring data science models into production?

With a yearly turnover of 400 mil €, Bakkersland is by far the largest bakery in the Netherlands. Early every morning, the organization's 300 trucks deliver approximately 2 mil fresh baked breads to 1,200 supermarkets. To predict the consumer demand per-day per-retailer, Bakkersland asked GoDataDriven to develop a predictive sales planning model. This machine learning model optimizes the availability of fresh bread products while at the same time minimizes left-over. The sales forecast enables Bakkersland to produce more efficiently and plan further ahead.