Goals behind Data Conveyer Design

The idea behind Data Conveyer stemmed from involvement in numerous data-centric projects and realization that they generally follow the same pattern differing only in some “variable elements”. These elements, once identified, became configurable settings to be integrated during orchestration of a uniform data migration process. To support such flexibility, Data Conveyer features a powerful API (Application Programming Interface) that consists of over a 100 of configuration settings and dozens of user-definable functions.

To ease complexities of its API, Data Conveyer employs a comprehensive logic to assume default values and actions whenever configuration settings, functions or parameters are not supplied. In fact, the entire Data Conveyer process can be orchestrated and executed in a single-line statement. While more code is required in non-trivial cases, the settings and types exposed by Data Conveyer are designed to be intuitive and naturally fall into place.

Since introduction of functional programming features into .NET (such as LINQ, Task Parallel Library, Dataflow, and many others), it became possible to succinctly write powerful code implementing functionality that was previously only attainable via complex, multi-tier software. This power comes at a price of a relatively steep learning curve. Data Conveyer aims at leveraging all power of modern .NET features while interacting with its clients using a familiar object-oriented paradigm.

Related to the previous item is the overall efficiency of Data Conveyer based solutions. In contrast to traditional approaches, there is no need for dedicated infrastructure, such as database or application servers. Instead, solutions run in memory relying heavily on parallelism, concurrency and asynchronous processing. To minimize memory footprint, data is subjected to rigorous buffering as it passes through Data Conveyer.

In its early stages, Data Conveyer has undergone major refactoring episodes, which in fact included a complete rewrite at one point. The main motivation behind these efforts was desire to facilitate future product expansions. The current architecture of Data Conveyer makes it straightforward to introduce support for new data formats, new types of transformation as well as other features, for example logging.

Data Conveyer is a standalone executable. As such, it gets deployed by a simple file copy, regardless if for development or production purposes. There are no dependencies to load (aside from .NET), no Registry updates, no installers or uninstallers to run. Data Conveyer can also be deployed as an industry standard NuGet package.

Data Conveyer can be deployed on any platform that supports .NET Standard 2.0. This means that it is not Windows specific, but can apply to Linux, Mac OS and other systems as well. What’s more, with a new Blazor technology, a Data Conveyer based solution can be compiled into a WebAssembly and as such reside inside a web browser. Please refer to Sample Applications for an example of such solution.

From the outset, Data Conveyer has been subjected to stringent quality assurance measures. The product’s architecture very nicely fits the Test-Driven Development (TDD) paradigm, where sample input data is supplied to the test and the output data is asserted on. Every build of Data Conveyer involves a series of over 400 test cases containing a total of more than 6,000 assertions.