View source

{{experimental}}
== Description ==
The Data Reader is an external library function that allows the user to read data from tabular files (see [[#Supported File Formats|File Formats]]) to be incorporated as variables of ASCEND models. The syntax allows the user to specify the file location, format and select an arbitrary number of variables to be imported from the data, given an independent variable as a parameter, such as time. The normal intended application for the data reader was for reading weather data, so that simulations of solar energy systems could be performed in a similar way to that provided by TRNSYS (see [[Other modelling tools]]).

== Supported File Formats ==
At this point, the following file formats are supported:
* TMY2 weather data files, (as [http://rredc.nrel.gov/solar/old_data/nsrdb/tmy2/ described] by NREL)
* ACDB Australian Climate Data Bank files (see {{src|models/johnpye/datareader/acdb.c}} for more information).
* CSV comma separated values.

Limitations apply to the CSV-formatted files:
* The '''first column must contain your independent variable, in base SI units'''. For example, this might be time in seconds, or temperature in Kelvin.
* Each line in the file is currently limited to 9999 characters, including commas. 
* Independent variable values must be sorted and monotonically increasing.

The [[#See_also|See also]] section contains a link to an example model using a CSV data file.

== Supported Interpolation Methods ==
For a given simulation, it is possible that the independent variable can be situated between data points. For example, if the data in the file has been sampled at hourly intervals and the time step is one minute.

The Data Reader external function supports the following interpolation methods, to fill these gaps:
* Linear interpolation
* Constrained cubic spline interpolation ([http://www.korf.co.uk/util_2.html more information])

By default, the cubic spline method is used, due to its overall smoothness, especially in the vicinity of a data sample. See more information about usage in the [[#Parameters|Parameters Syntax]]

== Syntax and Usage ==
A short description of the Data Reader usage will be presented using an existing example in the file {{srcbranch|jose|models/johnpye/datareader/testtmy.a4c}}.
=== Importing the Data Reader Module ===
In the first instance, the Data Reader Module has to be imported into ASCEND. This is simply done by adding this line to the preliminary section of your ASCEND model.
<source lang=a4c>
IMPORT "johnpye/datareader/datareader";
</source>
In the case of the example this is done in line 4.
=== Configuring the Input parameters of the file ===

In order to link the Data Reader function with your file, a file name has to be provided, along with a supported file format and other parameters. This is done by creating a simple model that only contains text strings and can be referenced by the main model that incorporates the Data Reader.
<source lang=a4c>
MODEL drconf;
   filename IS_A symbol_constant;
   filename :== "johnpye/datareader/23161.tm2";
   format IS_A symbol_constant;
   format :== 'TMY2';
   parameters IS_A symbol_constant;
   parameters :== '2:linear,2:cubic,2:default';
END drconf;
</source>
The variables in this model have to be named exactly 'filename', 'format', and 'parameters' for the Data Reader to be able to pick them up. In this example it is shown that assigning a value to each one of the symbol constants at this stage is optional.

====Filename====

The filename of the data file can be declared by assigning the file path value to the filename variable of the <tt>drconf</tt> model.
<source lang=a4c>
filename :== "johnpye/datareader/23161.tm2";
</source>
In this example, the path is declared as a relative path to the ASCEND model library. If the data file does not reside within this directory, it is also possible to declare an absolute file path.

====Format====

The format that the file has been written in is declared by assigning the format name (e.g 'TMY2', 'ACDB') to the <tt>format</tt> variable of the <tt>drconf</tt> model.

====Parameters====

The <tt>parameters</tt> variable is used to declare which columns are to be read and assigned to which variables in your model. For example, a user would like to import 3 variables out of 20 variables available in a data file. Furthermore, it is required that these variables to be the 1st, 7th and 9th columns of the data file. This is performed by assigning the following string to the parameters variable of <tt>drconf</tt>:
<source lang=a4c>
parameters :=='1,7,9';
</source>
By listing these column numbers, separated by commas, the user is specifying that the first model variable is the 1st column of the data file, the second model variable is the 7th and the third model variable is the 9th column. The model variables are as per the section [[#Declaring Inputs and Outpout Variables|Declaring Variables]].

If the columns in the parameters string are in a different order, or even repeated, that is the way that they will be assigned to the variables. As per the TMY2 example, the same column 2 is assigned to three different declared variables.

If the user declares less column assignments, the remaining column assignments will be filled with default numbering starting from the first column.

The user can also specify that the values are evaluated using one of a selection of interpolation methods. For each data column, the user can specify the interpolation method used with a colon, as follows:
<source lang=a4c>
parameters :== '2:linear,2:cubic,2:default';
</source>
In this example, the user requires that the first variable is the second column of the data file, interpolated linearly, the second variable is the second data column, using the constrained cubic spline algorithm and the third variable is again the second data column, using the default interpolation algorithm. By default, the cubic algorithm is used, and this is indicated either by specifying 'default' or nor specifying an interpolation algorithm as before.

===Declaring Inputs and Outpout Variables===
Once the datareader has been configured, it can be incorporated into the main model in ASCEND. This is done by creating an instance of the <tt>drconf</tt> type as part of the main model:
<source lang=a4c>
tmydata IS_A drconf;
</source>
This allows the main model to pass all the necessary parameters to to the Data Reader. Every time ASCEND is required to solve the model, it can retrieve the values of these variables first, aiding the solving process of the other model variables.

The final step is to indicate to ASCEND which variables are input variables and which are output variables. Usually a single variable is needed to retrieve a set of values (e.g. time). Currently the Data Reader only supports one independent variable. On the other hand, the number of outputs depends only on the data file, and any number of outputs can be declared, provided they are less or equal to the number of columns in the data file. The following example illustrates this.
<source lang=a4c>
my_solar_data:datareader(
   t : INPUT;
   Gbnl,Gbnc,Gbns :OUTPUT;
   tmydata : DATA
   );
</source>
For this example to work 't','Gbnl', 'Gbnc' and 'Gbns' must have been previously declared as the main model variables, with statements such as 't IS_A time;'. In the example, 't' has been declared as an input. 'Gbnl', 'Gbnc' and 'Gbns' have been declared as outputs, and the instance of the Data Reader link in the main model 'tmydata' has been declared as containing additional data.

The total number of OUTPUT variables in this declaration must not exceed the maximum number of columns available in the data file. In most cases, there will be an [http://en.wikipedia.org/wiki/Injective_function injective](i.e. one to one) relationship between data columns and model variables and this prevents declaring more variables than columns.

==Further Work==
We plan to take this further by connecting the data reader with a sun position algorithm, because it is important when interpolating values of solar flux to take into accord the time when the sun rises and sets (see the {{src|models/johnpye/sunpos.a4c}} model file in the [[ModelLibrary]]). This algorithm will become available as an interpolation option.

This code is still under development. In particular, use of this code exposed a limitation in [[IDA]] when 'integrating' models that don't have any derivatives in them.

==See also==
* The {{srcdir|models/johnpye/datareader/}} directory in the [[ModelLibrary]].
* Recent improvements by [[Jose Zapata]] in {{srcbranch|jose|models/johnpye/datareader/}}
* A CSV file example {{srcbranch|jose|models/johnpye/datareader/testcsv.a4c}}
* [[Other modelling tools#TRNSYS|TRNSYS]].