RaqForum 22 No.
1 Reply  •  483 View  • 
     ...

Spring Cloud integrate SPL to implement microservices

 

Spring Cloud is an excellent framework for micorservices. Microservices can quickly be built based on Spring Cloud; esProc SPL (hereinafter referred to as SPL) is an excellent lightweight computing engine, complex data processing can be easily implemented based on SPL. The combination of both can well meet the data processing needs under the microservice scenario. This article describes how to integrate SPL in Spring Cloud to implement microservice data processing.

 

Set up the environment for Spring Cloud

Spring Cloud can be installed by downloading from the official website or by using an IDE downloading plug-in, which can be found in public resources and will not be covered here. Spring Cloud has many components, among which Eureka is mainly used for service governance, service registration, service discovery, etc. . When SPL is working with microservices, Eureka is mainly used for service publishing, other components that have little to do with data processing, such as configuration, security, fault tolerance, etc. , will not be mentioned here.

 

Setting up a Spring Cloud environment also requires installing and configuring Maven (which helps to do more with less). Again, further details will be spared here. Please configure it on your own.

 

In this article,the setup environment is IntelliJ IDEA, and other development tools may take slightly different steps, please note.

 

Establish registry

Create a new Maven project

Create the Eureka Server

Add a module

 

Select Spring Initiaizr, and the Java version should be selected correctly

 

Select Eureka Server under Spring Cloud Discovery and click Finish

 

At this point, the following dependencies will appear in pom

Add configuration

Add registry configuration, modify the application.yml file under the resources path (rename it to yml if under resources is application. Porperties).

server:
  port: 8761

eureka:
  instance:
    hostname: localhost
  client:
    registerWithEureka: false
    fetchRegistry: false
    serviceUrl:
      defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/

Add annotation

Add @EnableEurekaServer in the main program Application to declare this is the registry for the server.

 

Launch the Project (run Application.java), access this url in browser: http://localhost:8761/

At this point, no server has been registered in the registry.

 

Create a service provider

The steps for creating a service provider are exactly the same as the steps for the registry, creating a new module, and selecting the Eureka Server, which will not be covered here.

 

Modify the yml configuration:

server:
  port: 8762
eureka:
  instance:
    hostname: localhost
  client:
    serviceUrl:
      defaultZone: http://localhost:8761/eureka/ #website of registry
spring:
  application:
name: MyProvider

 

Add the annotation @EnableEurekaClient to the Application to declare self as a service provider.

Add test cases:

@RestController
@RequestMapping("/hello")
public String hello(String message) {
    return "Hello! I am Provider.I got your message:" + message;
}

 

Access in browser: http://localhost:8762/hello?message=hi,provider

 

Return to the registry, we can see that the service provider (myprovider) has been registered.

 

Next, SPL will be integrated in the service provider module to provide calculation services.

Integrate SPL

Prepare the jars of driver

SPL provides a standard JDBC driver for application to integrate and invoke that require three basic jar packages, which can be found in [installation directory]\esProc\lib.

esproc-bin-xxxx.jar  //esProc computing engine and JDBC drive pack
icu4j-60.3.jar  //internationalization 
jdom-1.1.3.jar  //parsing configuration file

 

Copy the above three jars to main-resources-lib under the service provider (create a new one if there is no lib directory).

Deploy raqsoftConfig.xml

raqsoftConfig.xml is the primary configuration file for SPL, in which information such as the home directory and addressing paths of SPL script are configured. Copy it to the application’s classpath.

 

Add dependencies

Add local jar dependencies in pom

        <dependency>
            <groupId>com.esproc</groupId>
            <artifactId>esproc-dm</artifactId>
            <version>3.1</version>
            <scope>system</scope>
            <systemPath>${project.basedir}/src/main/resources/lib/esproc-bin-20210811.jar</systemPath>
        </dependency>

        <dependency>
            <groupId>com.esproc</groupId>
            <artifactId>esproc-icu4j</artifactId>
            <version>60.3</version>
            <scope>system</scope>
            <systemPath>${project.basedir}/src/main/resources/lib/icu4j-60.3.jar</systemPath>
        </dependency>

        <dependency>
            <groupId>com.esproc</groupId>
            <artifactId>esproc-jdom</artifactId>
            <version>1.1.3</version>
            <scope>system</scope>
            <systemPath>${project.basedir}/src/main/resources/lib/jdom-1.1.3.jar</systemPath>
        </dependency>

 

Develop a use case

SPL has been integrated in after the above steps, and it is time to develop a SPL use case.

   RequestMapping(value="callSPL/{code}", method=RequestMethod.GET)
    public String callSPL(@PathVariable("code") String code) throws ClassNotFoundException, SQLException {
        Connection con = null;
        PreparedStatement st;
        ResultSet rs;
        String jsonResult = null;
        Class.forName("com.esproc.jdbc.InternalDriver");
        con = DriverManager.getConnection("jdbc:esproc:local://");
        System.out.println("code:"+code);
        //call dfxname(?,?,?)
        st = con.prepareCall("call stockRising(?)");
        st.setObject(1, code);
        st.execute();
        rs = st.getResultSet();
        if (rs.next()){
            jsonResult = rs.getObject(1).toString();
        }

        if (con != null) {
            con.close();
        }
        return jsonResult;
    }

 

The stockRising invoked here is the file name of SPL script (stockRising.dfx) , which calculates the number of days a stock has risen consecutively (the same share price is recorded as an increase). stockRising.dfx looks like this, and the script parameter is s_code

A

1

=T("StockRecords.txt").select(code==int(s_code))

2

=A1.sort(ddate)

3

=A2.group@i(price<price[-1]).new(code,~.len():risedays)

calculate the number of days the stock has risen consecutively

4

return json(A3)

output the result as a json string

The SPL script is interpreted and supports hot switching with changes taking effect in real time.

 

Launch the project, access in browser: http://localhost:8762/callSPL/110330

The details of the SPL execution results (consecutive increases of the stock symbol 110330) can be seen above. So far the service provider can provide external SPL calculation services.

 

Use the data source

In addition to reading file data, we can also connect to a data source in the SPL calculation service to complete the data calculation.

 

JDBC data sources

Add related configurations of database

Here is an example of connecting to MySQL. Add related dependencies of mysql driver in pom.

        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.44</version>
        </dependency>

 

Add a data source connection in raqsoftConfig.xml, and add the following configurations under the DBList node:

        <DBList>
            <DB name="mysql">
                <property name="url" value="jdbc:mysql://192.168.3.18:3306/test1?useCursorFetch=true"></property>
                <property name="driver" value="com.mysql.jdbc.Driver"></property>
                <property name="type" value="10"></property>
                <property name="user" value="root"></property>
                <property name="password" value="root"></property>
                <property name="batchSize" value="0"></property>
                <property name="autoConnect" value="true"></property>
                <property name="useSchema" value="false"></property>
                <property name="addTilde" value="false"></property>
                <property name="needTransContent" value="false"></property>
                <property name="needTransSentence" value="false"></property>
                <property name="caseSentence" value="false"></property>
            </DB>
        </DBList>

 

Service provider code:

@RequestMapping("/callSPL2")
    public String callSPL(String dfxName) throws ClassNotFoundException, SQLException {
        Connection con = null;
        PreparedStatement st;
        ResultSet rs;
        String jsonResult = null;

        Class.forName("com.esproc.jdbc.InternalDriver");
        con = DriverManager.getConnection("jdbc:esproc:local://");
        st = con.prepareCall("call "+dfxName+"()");	//take the dfx name as an argument
        st.execute();
        rs = st.getResultSet();

        if (rs.next()){
            jsonResult = rs.getObject(1).toString();
        }

        if (con != null) {
            con.close();
        }
        return jsonResult;
    }

 

Write the SPL script (studentsOrder.dfx) to calculate the ranking of students’ grades.

A

B

C

1

=mysql.query("select     ID,SchoolID,ClassName,Score  from   s_sestudent order by   ID").derive(ClassOrder,SchoolOrder,UnionOrder)

2

>A1.run(UnionOrder=A1.rank@z(~.Score,Score))

3

for   A1.group(SchoolID)

>A3.run(SchoolOrder=A3.rank@z(~.Score,Score))

4

for   A3.group(ClassName)

>B4.run(ClassOrder=B4.rank@z(~.Score,Score))

5

return   json(A1)

A5 returns the results in json format:

Access in browser: http://localhost:8762/callSPL2/studentsOrder, and invoke successfully.

 

Non-JDBC data sources

In addition to JDBC data sources, SPL also provides several non-JDBC data source access interfaces, here take MongoDB as an example to explain the process. Most of the non-JDBC interfaces for SPL are provided as external libraries, with slightly different configuration processes than JDBC data sources.

 

Prepare external library related jars

It can downloaded from: http://download.raqsoft.com.cn/esproc/ext/extlib/extlib-20210811.zip. MongoDB is primarily involved with the following jar packages.

Bson-4.3.0-SNAPSHOT.jar
mongodb-driver-core-4.3.0-SNAPSHOT.jar
mongodb-driver-legacy-4.3.0-SNAPSHOT.jar
mongodb-driver-sync-4.3.0-SNAPSHOT.jar
raq-mongo-cli-4.3.0.jar

Copy them to the specified directory (which can not be in the project) , for example: D:\software\raqsoft20210630\esProc\extlib\MongoCli.

 

Configure raqsoftConfig.xml

Add the external library configurations under the Esproc node

        <extLibsPath>D:\software\raqsoft20210630\esProc\extlib</extLibsPath>
	<importLibs>
	    <lib>MongoCli</lib> 
	</importLibs>

 

Prepare SPL script

A

1

=mongo_open("mongodb://127.0.0.1:27017/test")

connect to MongoDB

2

=mongo_shell(A1,"col1.find()").fetch()

query the set data

3

>mongo_close(A1)

close the connection

4

return json(A2)

The result of SPL query:

Access in browser: http://localhost:8762/callSPL2/mongo, and invoke successfully.

Encapsulate SPL

Furthermore, SPL can be encapsulated as a common method so that you can always use ONE method to invoke SPL to get calculation results when requesting services. This method needs to prepare the SPL script(DFX) only, which supports hot switching, allowing for full hot deployment of services.

 

Common method

Add a common method to the service provider that the name and parameters of the SPL script are passed through the service consumer, as follows:

   @RequestMapping(path="/callSPLCommonJson", method= RequestMethod.POST)
   public String callSPLCommonJson(@RequestParam String dfxName, @RequestParam String params) throws ClassNotFoundException, SQLException {
       System.out.println("dfxName->"+dfxName);
       System.out.println("params->"+params);
       Connection con = null;
       java.sql.PreparedStatement st;
       int colCount = 0;
       int i = 0;
       ResultSet rs;
       String param = "";
       String resultStr;
       LinkedHashMap<String, String> jsonMap = null;
       String jsonResult=null;

       Class.forName("com.esproc.jdbc.InternalDriver");
       con = DriverManager.getConnection("jdbc:esproc:local://");
       if(null==params || "".equals(params)){
           st = con.prepareCall("call " + dfxName + "()");
       }else{
           jsonMap= JSONObject.parseObject(params, new TypeReference<LinkedHashMap<String, String>>(){});
           for (Map.Entry<String, String> entry : jsonMap.entrySet()) {
               param += "?,";
           }
           if (param != null && !"".equals(param)) {
               int len = param.length();
               param = param.substring(0, len - 1);
           }
           //call dfxname(?,?,?)
           st = con.prepareCall("call " + dfxName + "(" + param + ")");

           for (Map.Entry<String, String> entry : jsonMap.entrySet()) {
               i++;
               st.setObject(i, entry.getValue());
           }
       }
       st.execute();
       rs = st.getResultSet();
       if (rs.next()){
           jsonResult = rs.getObject(1).toString();
       }
       if (con != null) {
           con.close();
       }
       return jsonResult;
   }

 

Here the parameters are submitted in the POST method (method= RequestMethod.POST), which involves two parameters:

@dfxName is the name of the SPL script (.dfx);

@params is the parameter of SPL script that needs to be organized in JSON format, for example:

{"code":"110330","b_date":"2009-01-02","e_date":"2009-01-25"}.

 

Prepare SPL script (stockRising2.dfx)

Calculate the consecutive increases of a stock over a period of time.

A

1

=T("StockRecords.txt").select(code==int(s_code)     && ddate>=date(b_date) && ddate<= date(e_date))

2

=A1.sort(ddate)

3

=A2.group@i(price<price[-1]).new(code,~.len():risedays)

4

return json(A3)

There are three parameters involved in the script, respectively the stock symbol (s_code), the query starting date (b_date), and the query ending date (e_date)

Use the Postman test to submit the post request, and the result is as follows:

 

Invoke SPL services

The SPL-encapsulated computing services described above use a standard REST interface and can therefore be invoked by any development language. Here are a few examples of calls.

 

Java

Invoke SPL microservices by Spring RestTemplate:

    MultiValueMap<String, Object> jsonMap = new LinkedMultiValueMap<String, Object>();
        jsonMap.add("dfxName","stockRising2");
        jsonMap.add("params","{\n    \"code\":\"110330\",\n    \"b_date\":\"2009-01-02\",\n    \"e_date\":\"2009-01-25\"\n   }");

        String result = restTemplate.postForObject("http://127.0.0.1:8762/callSPLCommonJson", jsonMap, String.class);

 

This can also be achieved through other methods such as HttpURLConnection, HttpClient, and so on.

 

Python

Invoke SPL microservices by Python requests:

key_dict = {
        "dfxName": "stockRising2",
        "params":
            "{"+
                    "\"code\":\"110330\","+
                    "\"b_date\":\"2009-01-02\","+
                    "\"e_date\":\"2009-01-26\""+
                    "}"
            }
            
response = requests.post("http://127.0.0.1:8762/callSPLCommonJson", params=key_dict, headers={'Content-Type': 'application/json'})

 

SPL

SPL microservices can also be invoked by SPL:

A

1

stockRising2

2

{"code":"110330","b_date":"2009-01-02","e_date":"2009-01-26"}

3

=httpfile("http://127.0.0.1:8762/callSPLCommonJson","dfxName="+A1+"&params="+urlencode(json(A2),"utf-8")).read()
 
 •  483 View  •