Data Conveyer Features

General

  • Built-in support for various data formats including:
    • Delimited values (e.g. CSV)
    • Fixed width fields (flat file),
    • Key-value pairs (keywords)
    • XML
    • JSON
    • EDI/X12 (e.g. HIPAA transactions).
  • Support for strongly typed data elements.
  • Data buffering throughout the process to minimize memory usage.
  • Operation in-progress is cancelable.
  • Progress reporting.

Intake

  • Connectivity to any data source, real-time or offline; either synchronous or asynchronous.
  • Support for multiple sources.
  • Variety of settings to fine tune data parsing process.
  • Ability to combine related records into so-called clusters.

Transformation

  • Parallel processing (multiple engines).
  • Support for various transformation modes, including per record and per cluster.
  • Flexible API allowing different ways to manipulate data depending on circumstances and individual preferences.

Output

  • Connectivity to any target, real-time or offline; either synchronous or asynchronous.
  • Support for multiple targets.
  • Custom routing rules.
  • Variety of settings to fine tune data formatting process.

Planned for Future Releases

  • Build-in support for additional data formats, such as HL7 data.
  • Pipelined (cascaded) transformations.
  • Additional transformer types:
    • Aggregator
    • Sorter.

Goals behind Data Conveyer Design

The idea behind Data Conveyer stemmed from involvement in numerous data-centric projects and realization that they generally follow the same pattern differing only in some “variable elements”. These elements, once identified, became configurable settings to be integrated during orchestration of a uniform data migration process. To support such flexibility, Data Conveyer features a powerful API (Application Programming Interface) that consists of over a 100 of configuration settings and dozens of user-definable functions.

To ease complexities of its API, Data Conveyer employs a comprehensive logic to assume default values and actions whenever configuration settings, functions or parameters are not supplied. In fact, the entire Data Conveyer process can be orchestrated and executed in a single-line statement. While more code is required in non-trivial cases, the settings and types exposed by Data Conveyer are designed to be intuitive and naturally fall into place.

Since introduction of functional programming features into .NET (such as LINQ, Task Parallel Library, Dataflow, and many others), it became possible to succinctly write powerful code implementing functionality that was previously only attainable via complex, multi-tier software. This power comes at a price of a relatively steep learning curve. Data Conveyer aims at leveraging all power of modern .NET features while interacting with its clients using a familiar object-oriented paradigm.

Related to the previous item is the overall efficiency of Data Conveyer based solutions. In contrast to traditional approaches, there is no need for dedicated infrastructure, such as database or application servers. Instead, solutions run in memory relying heavily on parallelism, concurrency and asynchronous processing. To minimize memory footprint, data is subjected to rigorous buffering as it passes through Data Conveyer.

In its early stages, Data Conveyer has undergone major refactoring episodes, which in fact included a complete rewrite at one point. The main motivation behind these efforts was desire to facilitate future product expansions. The current architecture of Data Conveyer makes it straightforward to introduce support for new data formats, new types of transformation as well as other features, for example logging.

Data Conveyer is a standalone executable. As such, it gets deployed by a simple file copy, regardless if for development or production purposes. There are no dependencies to load (aside from .NET), no Registry updates, no installers or uninstallers to run. Data Conveyer can also be deployed as an industry standard NuGet package.

Data Conveyer can be deployed onto any platform that supports .NET Standard 2.0. This means that it is not Windows-specific, but equally applies to Linux, Mac OS and other systems as well. Solutions that involve Data Conveyer can be easily containerized, for example using Docker. What’s more, with a new Blazor technology, a Data Conveyer based solution can be compiled into a WebAssembly and as such reside inside a web browser. Please refer to Sample Applications for an example of such solution.

From the outset, Data Conveyer has been subjected to stringent quality assurance measures. The product’s architecture very nicely fits the Test-Driven Development (TDD) paradigm, where sample input data is supplied to the test and the output data is asserted on. Every build of Data Conveyer involves a series of over 400 test cases containing a total of more than 6,000 assertions.