Background and method

This routine is applicable to the following scenarios: data maintenance has no real-time requirements and can be performed regularly in a specific period (usually in hours or day); the total data is not large so that it can be stored in a single composite table; two modes, append and update, are supported, and the data amount for maintenance each time may be large, and may be passed in as a cursor.

Method: use a current composite table for query, and merge the received new data with the current composite table to generate a backup composite table. After merging, let the current composite table point to the newly generated backup composite table to provide external query services, and let the backup composite table point to the current composite table and delete when the next update starts.

Definitions and concepts

1. Current composite table and backup composite table: the current composite table refers to the composite table currently in use; the backup composite table refers to the current composite table that was discarded in the last update and will be deleted in the next update. When updating, merge the new data with the current composite table to write to the backup composite table. Just set the file name of the current composite table and backup composite table in config. After updating, exchange the current composite table and backup composite table in config, which is equivalent to discarding the original current composite table.

2. addOnly: refers to whether or not to append data only. If the value is false, it means not only to append data but also to modify/delete data.

Configuration file description

Configuration file: ubc.json

By default, this file is located in the main path of esProc (if you want to save it to another directory, modify the code to use an absolute path). The configuration file includes the following contents:

[{
"addOnly":true,
"sortKey":"account,tdate",
"deleteField":null,
"otherFields":"ColName1,ColName2",
"dataDir":"data/",
"dataFilename":"data.splx",
"backup":"data1.splx"
}]

“addOnly”: refers to whether or not to append data only, and its value is true/false, where false means update mode.

“sortKey” refers to the sorting field of composite table. If data update is involved, it also refers to the primary key field. When there are multiple sorting fields, they are separated by commas.

“deleteField” refers to the name of the deletion flag field. If there is no deletion operation, fill in null. The value of this field is true/false. If the value of addOnly is true, this parameter value can only be filled with null.

“otherFields” refers to other fields of composite table. If there are multiple other fields, they are separated by commas.

“dataDir” refers to the storage path of composite table relative to the main directory.

“dataFilename” refers to the name of the current composite table file, such as “data1.ctx”.

“backup” refers to the name of backup composite table file, such as “data1.ctx”.

Storage structure

Files and sub-paths in the main path:

data: refers to the storage path of composite table, and its name is set in ubc.json (refer to the description above).

ubc.json: configuration file

The files under ‘data’ include:

The name of composite table file can be set in ubc.json (refer to the description above).

Global variable

current: refers to the current composite table file.

Code analysis

init.splx

This script will be executed when the server starts. If the server is started for the first time, it needs to initialize parameters and create an initial composite table.

	A	B
1	>config=json(file("ubc.json").read())
2	=file(config.dataDir/config.dataFilename)
3	if(!A2.exists())
4		=config.sortKey.split@c().("#"+trim(~)).concat@c()
5		=if(config.deleteField && !config.addOnly, A2.create@d(${B4},${config.deleteField},${config.otherFields}), A2.create(${B4},${config.otherFields}) )
6		=B5.close()
7	>env(current,A2)

A1: read the configuration file;

A3-B5: if the current composite table file does not exist, create a new one;

B5: if there is a deletion flag field and the addOnly value is false, add a deletion flag field when creating the composite table file, and add the @d option;

A7: set the current composite table file as global variable.

update.splx

This script is called and executed by an external program, and to merge the received new data with the current composite table.

Input parameter: cs data cursor.

	A
1	>config=json(file("ubc.json").read())
2	=movefile(config.dataDir/config.backup)
3	=file(config.dataDir/config.dataFilename)
4	=file(config.dataDir/config.backup)
5	=A3.reset${if(config.addOnly,"","@w")}(A4;cs)
6	=config.dataFilename
7	>config.dataFilename=config.backup,current=file(config.dataDir/config.dataFilename)
8	>config.backup=A6
9	=file("ubc.json").write(json(config))

A1: read the configuration file;

A2: delete the backup composite table;

A3-A5: merge the current composite table and passed-in cursor data together, and then write them to the backup composite table;

A6-A8: exchange the backup composite table and current composite table, and update the global variable current.

query.splx

This script is used for data query and will return a composite file object.

	A
1	return current

Note: the composite table object must be closed after use.

Download the routine code: code.zip